From 4f05859cb81939ed969543f3d74638728207bd7d Mon Sep 17 00:00:00 2001 From: hubcio Date: Fri, 11 Sep 2026 00:16:49 +0200 Subject: [PATCH 01/13] fix(docs): align clustering with server 0.9.0 --- content/docs/clustering/client-failover.mdx | 18 +++++++------ content/docs/clustering/configuration.mdx | 28 +++++++++++++++----- content/docs/clustering/deploy.mdx | 21 ++++++++++----- content/docs/clustering/durability.mdx | 29 ++++++++++++--------- content/docs/clustering/security.mdx | 6 +++-- content/docs/clustering/vsr.mdx | 8 +++--- 6 files changed, 70 insertions(+), 40 deletions(-) diff --git a/content/docs/clustering/client-failover.mdx b/content/docs/clustering/client-failover.mdx index ebe9de48ee..ea86024e0f 100644 --- a/content/docs/clustering/client-failover.mdx +++ b/content/docs/clustering/client-failover.mdx @@ -7,28 +7,30 @@ A clustered deployment changes what a client can expect from any single node: re ## Where requests are served -**Reads** are served from the local replicated state on whichever node owns the namespace locally, followers included. A client connected to a follower can poll messages and read metadata without touching the primary. +**Reads** use the local replicated state by default on whichever node owns the namespace locally, followers included. A client connected to a follower can poll messages and read metadata without touching the primary. -**Writes** go through consensus and are admitted on the plane's primary *only*: metadata operations on the metadata primary, message and offset writes on the primary of that partition's consensus group. A write returns after the required VSR commit, or returns a retryable error when the replica is changing view or catching up. +**Writes** go through consensus and are admitted on the plane's primary *only*: metadata operations on the metadata primary, message and offset writes on the primary of that partition's consensus group. An acknowledged write returns after the required VSR commit, or returns a retryable error when the replica is changing view or catching up. HTTP `ack=none` returns after dispatch instead. -One caveat on follower reads: server-managed offset auto-commit replicates the polled offset through the partition consensus, which only the partition primary may do. A poll served by a follower **does not advance the durable consumer offset**. Auto-commit is best-effort *at-least-once* delivery either way, so this widens the redelivery window after failover rather than losing data. +One caveat on follower reads: server-managed offset auto-commit replicates the polled offset through the partition consensus, which only the partition primary may do. A poll served by a follower **does not advance the durable consumer offset**. Auto-commit is best-effort and the poll response does not await offset persistence. On a primary, the offset submission precedes delivery of the poll reply, so it does not guarantee that the application received or processed the batch. Store offsets explicitly after processing when that ordering is required. ## Leader redirection -Leader-aware SDKs find the primary instead of requiring you to point them at it: +Leader-aware SDKs discover the metadata primary. Rust binary clients follow this sequence: -1. The client connects to any node from its connection string and fetches cluster metadata, which lists every node with its name, client endpoints, role, and status. +1. The client connects and authenticates to its configured node, then fetches cluster metadata, which lists every node with its name, client endpoints, role, and status. 2. When the connected node isn't the leader, the client reconnects to the leader's advertised endpoint for the transport in use. -3. When the cluster is transiently leaderless (for roughly one heartbeat timeout after a primary fails, while the election completes), the client polls the metadata until a leader appears instead of failing. +3. When the cluster is transiently leaderless, the client polls metadata every 250ms for up to 5s. If no healthy leader appears within that window, it continues on the contacted node and lets subsequent requests report their outcome. 4. Redirects are **capped** (three in the Rust SDK) so a flapping roster can't bounce the client forever. The address a client is redirected to is the node's `advertised_address` (or the matching `advertised_addresses` selector), falling back to the roster `ip`. Getting those right matters: the client redials **exactly what the metadata advertises**. See [Configuration](/docs/clustering/configuration) for the selector rules. +Metadata and partition primaries can differ. After a `TransientNotAccepted` response, Rust binary clients check the metadata leader and can then try the remaining advertised endpoints in a bounded roster walk. This fallback is separate from the three-redirect limit. + The Rust implementation lives in [`core/sdk/src/leader_aware.rs`](https://github.com/apache/iggy/blob/master/core/sdk/src/leader_aware.rs). ## Retryable errors -During a view change or while a replica catches up, writes fail with transient errors rather than definitive ones. The SDKs replay those transparently, so a primary failover surfaces as **added latency rather than an error**, as long as it completes within the client's retry budget. Definitive errors (validation failures, permission denials) are **never replayed**. +During a view change or while a replica catches up, writes can return transient errors. Rust binary clients retry within their request budgets. `TransientNotAccepted` proves that the request was never admitted and permits retry on another node. `TransientNotCommitted` has an uncertain outcome and is replayed with the same session and request identity, then returned if the budget expires. Disconnects and exhausted routing attempts can still reach the caller. Definitive validation and permission errors are **not replayed**. ## SDK support @@ -50,4 +52,4 @@ Run the suite for one SDK with: ./scripts/run-bdd-tests.sh rust leader_redirection ``` -For SDKs without redirection coverage, connect clients to the current primary for write-heavy workloads, or front the cluster's HTTP transport with a load balancer and rely on [follower HTTP forwarding](/docs/clustering/security) for control-plane operations. +For SDKs without redirection coverage, connect clients to the current primary for write-heavy workloads, or front the cluster's HTTP transport with a load balancer and rely on [follower HTTP forwarding](/docs/clustering/security) for control-plane operations and acknowledged partition writes. diff --git a/content/docs/clustering/configuration.mdx b/content/docs/clustering/configuration.mdx index 1bb5acb001..aea592579f 100644 --- a/content/docs/clustering/configuration.mdx +++ b/content/docs/clustering/configuration.mdx @@ -5,7 +5,7 @@ description: Reference for the [cluster] settings, consensus timing, and node ro All clustering settings live in the `[cluster]` section of the server configuration. The authoritative defaults with detailed comments are in [`core/server/config.toml`](https://github.com/apache/iggy/blob/master/core/server/config.toml). Every value can also be set through an `IGGY_`-prefixed environment variable, for example `IGGY_CLUSTER_ENABLED=true` or `IGGY_CLUSTER_NODES_0_IP=10.0.1.5`. -The server validates the whole section at boot and **refuses to start** on an invalid value, printing the exact rule that failed. +With clustering enabled, the server validates the section at boot and **refuses to start** on an invalid value, printing the rule that failed. The repair chunk limit and superblock failure timeout are validated even when clustering is disabled. ## Top-level settings @@ -18,7 +18,7 @@ The server validates the whole section at boot and **refuses to start** on an in ## Consensus timing -Consensus runs on a **fixed 10ms tick**. Every duration below is converted to whole ticks: values are **rounded down** to a multiple of 10ms, and anything under 10ms is raised to a single tick rather than firing sooner. +Consensus runs on a **fixed 10ms tick**. Every duration below is converted to whole ticks: values are **rounded down** to a multiple of 10ms, and anything under 10ms is raised to a single tick rather than firing sooner. `repair_gap_debounce_interval` has an additional 500ms floor. | Setting | Default | Purpose | | --- | --- | --- | @@ -30,6 +30,7 @@ Consensus runs on a **fixed 10ms tick**. Every duration below is converted to wh | `request_start_view_retransmit_interval` | `1s` | How often a recovering replica re-requests the current view's `StartView` | | `view_probe_attempts_max` | `5` | Unanswered `RequestStartView` probes a recovering replica tolerates before falling back to an election | | `repair_retry_interval` | `1s` | How long a stalled journal-repair stream waits before re-requesting its remaining window | +| `repair_gap_debounce_interval` | `1s` | How long a committed-history gap waits before opening a repair session; floored at 500ms | | `repair_chunk_max` | `128` | Prepares a peer serves per repair round | Raise `heartbeat_timeout` on oversubscribed hosts where scheduling stalls fake primary death. Otherwise keep the defaults unless tests show a specific scheduling or network problem. @@ -44,9 +45,22 @@ Boot-time validation enforces these constraints: - `view_probe_attempts_max` must be between 1 and 100. - `repair_chunk_max` must be between 1 and 1024, and strictly below `message_bus.peer_queue_capacity`: each repair frame rides the per-peer message-bus queue, and a full round above that capacity would overrun the queue and drop frames. +## Superblock failure handling + +`superblock_wedged_fatal_timeout` defaults to `2m`. It controls how long repeated superblock persistence failures can continue before the process exits. The affected consensus group stays fenced while persistence is failing. `0`, `disabled`, or `unlimited` disables the process exit, leaving the group fenced; a nonzero value must be at least `30s`. + +## Coordinator placement + +These settings live under `[cluster.coordinator]` and apply when the server has more than one shard: + +| Setting | Default | Purpose | +| --- | --- | --- | +| `skip_shard_zero_for_replicas` | `true` | Places replica connections on peer shards instead of shard 0 | +| `skip_shard_zero_for_clients` | `false` | When enabled, excludes shard 0 from client connection placement | + ## Node roster -`cluster.nodes` is the full roster of cluster members and is intended to be **byte-identical on every node**. The running node finds its own entry through the `--replica-id` CLI flag. +`cluster.nodes` is the full roster of cluster members and is intended to be **byte-identical on every node**. The running node finds its own entry through the `--replica-id` CLI flag. Environment variables replace the entire `cluster.nodes` array rather than merging individual fields into TOML entries. If you configure any roster entry through environment variables, provide the complete roster that way. ```toml [[cluster.nodes]] @@ -63,7 +77,7 @@ Node name, unique within the roster and non-empty. ### `ip` -The node's roster address. Replica-to-replica consensus traffic and follower-to-primary HTTP forwarding dial it, so it must be a **literal IPv4 or IPv6 address**. Hostnames are rejected when the server resolves the roster at startup. +The node's roster address. Replica-to-replica consensus traffic and follower-to-primary HTTP forwarding dial it, so it must be a **literal IPv4 or IPv6 address**. Hostnames and unspecified addresses (`0.0.0.0` or `::`) are rejected when the server resolves the roster at startup. `ip` is **not the bind interface** for the client transports. `tcp`, `quic`, `http`, and `websocket` bind whatever their own `address` settings say. The roster supplies only their ports. The defaults bind loopback, so a cluster spread across hosts needs each transport's `address` set to `0.0.0.0` or the routable interface. The server warns at startup when a bind cannot serve the advertised `ip`. @@ -73,7 +87,7 @@ Numeric replica id for VSR consensus, `0`-based. Ids must be unique and strictly ### `ports` -Per-node listener ports: `tcp`, `quic`, `http`, `websocket`, and `tcp_replica`. In cluster mode `ports` is the **single source of listener ports**: every enabled transport needs an explicit per-node port, otherwise the server refuses to start. `tcp_replica` carries consensus traffic and is always required. Port `0` is rejected, and no two roster entries may claim the same `ip:port`. +Per-node listener ports: `tcp`, `quic`, `http`, `websocket`, and `tcp_replica`. In cluster mode `ports` is the **single source of listener ports**: every enabled transport needs an explicit per-node port, otherwise the server refuses to start. `tcp` and `tcp_replica` are always required, even when client TCP is disabled. `tcp_replica` carries consensus traffic. Port `0` is rejected, and no two roster entries may claim the same `ip:port`. ### `advertised_address` @@ -89,7 +103,7 @@ name = "iggy-node-1" ip = "10.0.1.5" # replica plane + last-resort fallback advertised_address = "203.0.113.10" # catch-all for unmatched clients replica_id = 0 -ports = { tcp = 8090, http = 3000, tcp_replica = 9090 } +ports = { tcp = 8090, quic = 8080, http = 3000, websocket = 8093, tcp_replica = 9090 } [[cluster.nodes.advertised_addresses]] client_cidr = "10.0.0.0/16" # in-VPC clients stay private @@ -105,7 +119,7 @@ Rules and caveats: - Matching sees the transport-level peer address, so clients behind a proxy or load balancer match the proxy's network, not their own. - Every `address` must be routable from inside its own `client_cidr`: leader-aware clients redial whatever address metadata advertises, and a selector pointing at an unreachable host strands them mid-redirect. - Prefer literal IPs over hostnames. SDKs differ in how they compare an advertised hostname against the address they dialed, and a mismatch costs a reconnect on every fresh connect. -- Rolling upgrades: older server binaries reject a TOML config containing `advertised_addresses` but **silently ignore** the equivalent `IGGY_CLUSTER_NODES_*_ADVERTISED_ADDRESSES_*` env vars. Upgrade every binary first, then add selectors. +- Rolling upgrades: VSR builds predating selector support reject a TOML config containing `advertised_addresses`. Their release builds warn and ignore the equivalent `IGGY_CLUSTER_NODES_*_ADVERTISED_ADDRESSES_*` env vars; debug builds can panic on them. Upgrade every binary first, then add selectors. ## Authentication and TLS diff --git a/content/docs/clustering/deploy.mdx b/content/docs/clustering/deploy.mdx index 72d0d8c128..2d7afeb7b0 100644 --- a/content/docs/clustering/deploy.mdx +++ b/content/docs/clustering/deploy.mdx @@ -30,7 +30,7 @@ The default `core/server/config.toml` has clustering disabled, so a plain run st cargo run --bin iggy-server -- --with-default-root-credentials ``` -`--with-default-root-credentials` sets the root user to `iggy`/`iggy` unless `IGGY_ROOT_USERNAME` and `IGGY_ROOT_PASSWORD` are already exported. **Only the first boot** on an empty data directory reads these values. +`--with-default-root-credentials` sets the root user to `iggy`/`iggy` unless `IGGY_ROOT_USERNAME` and `IGGY_ROOT_PASSWORD` are already exported. These values initialize the root user **only on its first creation**. Later starts recover the stored user; supplied environment values are still validated at boot. When clustering is disabled, `--replica-id 0` is still accepted. Any other id is rejected because it would have to match a `cluster.nodes` entry. @@ -79,7 +79,7 @@ export IGGY_ROOT_PASSWORD=iggy export IGGY_CLUSTER_AUTH_SHARED_SECRET="replace-with-at-least-32-random-bytes" ``` -Root credentials are **mandatory** once the cluster is enabled. The shared secret must be **at least 32 bytes**. See [Security](/docs/clustering/security) for how it's used and rotated. +Root credentials are **mandatory on the first boot of each replica** when clustering is enabled. Restarts recover the stored root user. The shared secret must be **at least 32 bytes**. See [Security](/docs/clustering/security) for how it's used and rotated. Start each replica in a separate terminal. Use a different data path for every process: @@ -101,7 +101,7 @@ In cluster mode: - `--replica-id` is required and must match exactly one `cluster.nodes` entry - `replica_id` values must be unique and contiguous from `0` - `ports` is the single source of listener ports: every enabled transport needs an explicit per-node port, otherwise the server **refuses to start** -- `tcp_replica` carries replica-to-replica consensus traffic and is **always required** +- `tcp` and `tcp_replica` ports are **always required**, even when client TCP is disabled; `tcp_replica` carries replica-to-replica consensus traffic - `ip` must be a **literal IP address**. Use `advertised_address` when clients can't reach it (see [Configuration](/docs/clustering/configuration)) - use a different root `path` for each process on the same host @@ -126,7 +126,16 @@ IGGY_CLUSTER_NODES_0_PORTS_WEBSOCKET=8093 IGGY_CLUSTER_NODES_0_PORTS_TCP_REPLICA=9090 IGGY_CLUSTER_NODES_1_NAME=node-2 IGGY_CLUSTER_NODES_1_IP=172.28.0.102 -# ... and so on for every node and every enabled transport +IGGY_CLUSTER_NODES_1_REPLICA_ID=1 +IGGY_CLUSTER_NODES_1_PORTS_TCP=8090 +IGGY_CLUSTER_NODES_1_PORTS_QUIC=8080 +IGGY_CLUSTER_NODES_1_PORTS_HTTP=3000 +IGGY_CLUSTER_NODES_1_PORTS_WEBSOCKET=8093 +IGGY_CLUSTER_NODES_1_PORTS_TCP_REPLICA=9090 +IGGY_TCP_ADDRESS=0.0.0.0:8090 +IGGY_QUIC_ADDRESS=0.0.0.0:8080 +IGGY_HTTP_ADDRESS=0.0.0.0:3000 +IGGY_WEBSOCKET_ADDRESS=0.0.0.0:8093 IGGY_CLUSTER_AUTH_ENABLED=true IGGY_CLUSTER_AUTH_SHARED_SECRET=replace-with-at-least-32-random-bytes IGGY_ROOT_USERNAME=iggy @@ -155,7 +164,7 @@ The response lists the cluster name, nodes, client endpoints, and current roles. ## Run the cluster test suites -The cross-SDK BDD suites run against a real cluster started from `bdd/docker-compose.cluster.yml`: +The cross-SDK BDD suites use a standalone server for basic features and add the two-node cluster from `bdd/docker-compose.cluster.yml` for `leader_redirection` or `all`: ```bash ./scripts/run-bdd-tests.sh rust @@ -163,4 +172,4 @@ The cross-SDK BDD suites run against a real cluster started from `bdd/docker-com ./scripts/run-bdd-tests.sh rust leader_redirection ``` -The first argument selects the SDK (`rust`, `python`, `php`, `go`, `go-race`, `node`, `csharp`, `java`, `cpp`, or `all`) and the optional second argument selects one feature (`basic_messaging`, `leader_redirection`, `raw_command`, or `all`). `leader_redirection` is the clustered feature. The Rust, Go, C#, and Java suites support it. +The first argument selects the SDK (`rust`, `python`, `php`, `go`, `go-race`, `node`, `csharp`, `java`, `cpp`, or `all`) and the optional second argument selects one feature (`basic_messaging`, `leader_redirection`, `raw_command`, `stream_crud`, or `all`). `leader_redirection` is the clustered feature. The Rust, Go, C#, and Java suites support it. diff --git a/content/docs/clustering/durability.mdx b/content/docs/clustering/durability.mdx index ec1c261e65..479df2edb0 100644 --- a/content/docs/clustering/durability.mdx +++ b/content/docs/clustering/durability.mdx @@ -39,7 +39,7 @@ Iggy's replication quorum is not a strict majority for every group size: Except for two replicas, the replication quorum is `min(ceil(n / 2), 3)` and the view-change quorum is `n - replication_quorum + 1`. Two replicas require both for either quorum. The quorums therefore intersect. These are -the [implemented quorum rules](https://github.com/apache/iggy/blob/97f7b0c0335f81691bef34f923f21d783016a6a4/core/consensus/src/impls.rs), +the [implemented quorum rules](https://github.com/apache/iggy/blob/4a38798a506c9231afdfc83a34ebed394d41dac2/core/consensus/src/impls.rs), not a configurable acknowledgement count. ## Replicated completion @@ -63,15 +63,17 @@ Replication count alone does not establish a bound on that loss. ## Persisted completion When either policy is `persisted`, each multi-replica partition uses a -bounded on-disk prepare WAL. The WAL records full prepares, including message -payloads. A prepare requiring persistence cannot release its `PrepareOk` -until its history and durable frontier are recoverable. +bounded on-disk prepare WAL. Message prepares contain headers and segment +references; their payloads live in segment files retained by hard links until +the WAL history can be reclaimed. Other operations are stored inline. A prepare +requiring persistence cannot release its `PrepareOk` until its message bodies, +WAL history, and durable frontier are recoverable. Prepares can be forwarded while local persistence is pending. Once enough replicas have met the required barrier, the operation can commit and the -primary can apply it and reply. An acknowledged message may still be in -the prepare WAL rather than in a segment file; its recovery does not depend -on first reaching a segment flush threshold. +primary can apply it and reply. An acknowledged message can be recovered +through the WAL before ordinary segment materialization and index updates +finish; recovery does not depend on first reaching a segment flush threshold. The WAL also retains predecessors across message and offset operations. Consequently: @@ -79,7 +81,7 @@ Consequently: - `durability=persisted` does not make an offset response persisted when `consumer_offset_durability=replicated`. - `consumer_offset_durability=persisted` with replicated messages still - journals message payloads. A durable offset's predecessor history must + retains message payloads through WAL segment references. A durable offset's predecessor history must remain recoverable. - A shared barrier can also persist co-batched operations with the weaker policy. That incidental persistence does not strengthen what their earlier @@ -89,8 +91,9 @@ Consequently: WAL history is reclaimed only after the materialized segment and offset state needed to replace it has been synchronized. The server's -`[partition] wal_bytes_max` setting, default `256 MiB`, bounds active WAL -and queued/in-flight prepare bytes per partition. Capacity pressure causes +`[partition] wal_bytes_max` setting, default `256 MiB`, bounds retained +and queued/in-flight prepare bytes per partition. The accounting includes +message bodies even when the WAL stores only references to them. Capacity pressure causes checkpointing and backpressure; it does not downgrade a persisted operation. Temporary rewrites need additional disk space. @@ -130,8 +133,8 @@ Neither policy turns HTTP `ack=none` or a poll's auto-commit into an awaited durable result. See [Which responses prove completion](/docs/server/durability#which-responses-prove-completion). The storage mechanisms are implemented in the -[partition persistence worker](https://github.com/apache/iggy/blob/97f7b0c0335f81691bef34f923f21d783016a6a4/core/partitions/src/persistence.rs) -and [prepare journal](https://github.com/apache/iggy/blob/97f7b0c0335f81691bef34f923f21d783016a6a4/core/journal/src/partition_journal.rs). -The [crash-recovery tests](https://github.com/apache/iggy/blob/97f7b0c0335f81691bef34f923f21d783016a6a4/core/integration/tests/cluster/crash_durability.rs) +[partition persistence worker](https://github.com/apache/iggy/blob/4a38798a506c9231afdfc83a34ebed394d41dac2/core/partitions/src/persistence.rs) +and [prepare journal](https://github.com/apache/iggy/blob/4a38798a506c9231afdfc83a34ebed394d41dac2/core/journal/src/partition_journal.rs). +The [crash-recovery tests](https://github.com/apache/iggy/blob/4a38798a506c9231afdfc83a34ebed394d41dac2/core/integration/tests/cluster/crash_durability.rs) exercise acknowledged persisted messages and offsets below ordinary flush thresholds, on both a singleton and a three-replica cluster. diff --git a/content/docs/clustering/security.mdx b/content/docs/clustering/security.mdx index 7391a382e9..d66882e83c 100644 --- a/content/docs/clustering/security.mdx +++ b/content/docs/clustering/security.mdx @@ -62,6 +62,8 @@ Follower nodes forward control-plane HTTP requests (stream, topic, user, and sim - explicitly configured `http.jwt` signing secrets, identical on every node, or - the cluster PSK: when HTTP is enabled, cluster auth is enabled, and no `http.jwt` secrets are configured, the signing key is derived from `shared_secret`. -Either way, a bearer token minted on any node verifies on every node, which is the invariant the forwarding depends on. Forwarded requests dial the primary at its roster `ip` and `ports.http`. The follower verifies the caller's bearer locally before forwarding, and the primary re-authenticates the request through its normal stack. +Either way, matching JWT key material lets a bearer token minted on one node verify on another. If JWT keys are derived from the PSK, rotating the PSK also changes the JWT key: `previous_shared_secret` applies only to replica authentication. Use independently configured, shared JWT keys when HTTP tokens must remain valid across PSK rotation. Forwarded requests dial the primary at its roster `ip` and `ports.http`. The follower verifies the caller's bearer locally before forwarding, and the primary re-authenticates the request through its normal stack. -The forwarding covers the control plane only. Partition-plane writes over HTTP (producing messages, storing consumer offsets) are **not forwarded**: each partition is its own consensus group whose primary can diverge from the metadata primary, so those requests must land on the right node. See [Client failover](/docs/clustering/client-failover) for how clients find it. +Acknowledged partition-plane writes over HTTP (producing messages, storing or deleting consumer offsets) first run on the contacted node. If it returns `TransientNotAccepted`, the server tries the other configured HTTP nodes at most once each, within a bounded retry window. That response proves the operation was never admitted. Ambiguous outcomes are returned without replay, and `ack=none` does not provide an acknowledgement for this fallback. Each partition is its own consensus group, so its primary can differ from the metadata primary. See [Client failover](/docs/clustering/client-failover) for how clients find it. + +When HTTP TLS is enabled, forwarding uses HTTPS and pins the destination certificate to the forwarding node's own HTTP leaf certificate. All nodes therefore need the same HTTP certificate for forwarding; distinct per-node HTTP certificates fail verification. Replica TLS is configured separately and does not encrypt the HTTP forwarding hop. diff --git a/content/docs/clustering/vsr.mdx b/content/docs/clustering/vsr.mdx index bfc89051c3..24ec835962 100644 --- a/content/docs/clustering/vsr.mdx +++ b/content/docs/clustering/vsr.mdx @@ -56,23 +56,23 @@ VSR uses three main flows: 2. **View change**: replicas exchange `StartViewChange` and `DoViewChange`, then the new primary sends `StartView`. 3. **Recovery**: a restarted or lagging replica requests the current view and repairs missing WAL ranges before serving current state. -A cluster of `2f + 1` replicas tolerates `f` unavailable replicas. Use **at least three replicas** for one-node fault tolerance. A two-node cluster is useful for development, but it **cannot make progress** after either node fails. +Three replicas tolerate one unavailable replica; five tolerate two. Larger groups use a capped replication quorum and a growing view-change quorum, so the usual `2f + 1` rule does not apply to every group size. See [Cluster durability](/docs/clustering/durability#what-an-acknowledgement-means) for the quorum rules. Use **at least three replicas** for one-node fault tolerance. A two-node cluster is useful for development, but it **cannot make progress** after either node fails. ## Wire protocol and SDKs -VSR framing is the *only* wire protocol. Every SDK (Rust, Go, Java, C#, C++, Node.js, Python) speaks it, and every BDD suite runs against the clustered-capable `iggy-server`. Clients built before the VSR migration **cannot talk to current servers**. +VSR framing is the *only binary* wire protocol, used by TCP, QUIC, and WebSocket connections. HTTP uses the REST API. Every SDK (Rust, Go, Java, C#, C++, Node.js, Python) speaks it, and every BDD suite runs against the clustered-capable `iggy-server`. Binary clients built before the VSR migration **cannot talk to current servers**. The Rust SDK needs no feature flags. Depend on the published crate: ```toml [dependencies] -iggy = "0.11.0-edge.4" +iggy = "0.11.0-edge.7" tokio = { version = "1", features = ["full"] } ``` or track the repository directly with `iggy = { git = "https://github.com/apache/iggy" }`. -The client API is the same in single-node and cluster mode: +The client API is the same in single-node and cluster mode. This example uses the `iggy` / `iggy` credentials explicitly provisioned in [Getting started](/docs/introduction/getting-started): ```rust use iggy::prelude::*; From 3dd207a500884e6783318f7ce9c4ede1a1dc0c8f Mon Sep 17 00:00:00 2001 From: hubcio Date: Fri, 11 Sep 2026 00:47:51 +0200 Subject: [PATCH 02/13] fix(docs): align introduction with server 0.9.0 --- content/docs/introduction/about.mdx | 46 ++++----- content/docs/introduction/architecture.mdx | 37 ++++--- content/docs/introduction/concepts.mdx | 34 +++---- content/docs/introduction/getting-started.mdx | 96 +++++++++++++------ src/components/architecture-diagrams.tsx | 36 +++---- 5 files changed, 146 insertions(+), 103 deletions(-) diff --git a/content/docs/introduction/about.mdx b/content/docs/introduction/about.mdx index 86531739f4..9c8129b438 100644 --- a/content/docs/introduction/about.mdx +++ b/content/docs/introduction/about.mdx @@ -15,6 +15,8 @@ The name is an abbreviation for the Italian Greyhound - small yet extremely fast ![Iggy Server](/img/iggy_server.png) +Historical startup screenshot from server 0.5.0. See [configuration](/docs/server/configuration) for the 0.9.0 settings. + --- ### Features @@ -31,29 +33,29 @@ The name is an abbreviation for the Italian Greyhound - small yet extremely fast - Available client SDK in multiple languages - **Works directly with binary data**, avoiding enforced schema and serialization/deserialization overhead - Custom **zero-copy (de)serialization**, which greatly improves the performance and reduces memory usage -- **Custom memory pool** with 28 buckets (buffer sizes from 4 KiB to 512 MiB) for pre-allocated, zero-copy message passing -- Configurable server features (e.g. caching, segment size, data flush interval, transport protocols etc.) +- **Custom memory pool** with 28 buckets (buffer sizes from 4 KiB to 512 MiB) for on-demand allocation, buffer reuse and sharing without copying buffer contents +- Configurable server features (e.g. caching and transport protocols), plus per-topic segment size, durability and flush thresholds - Server-side storage of **consumer offsets** - Multiple ways of polling the messages: - By offset (using the indexes) - By timestamp (using the time indexes) - First/Last N messages - Next N messages for the specific consumer -- Possibility of **auto committing the offset** (e.g. to achieve *at-most-once* delivery) -- **Consumer groups** providing the message ordering and horizontal scaling across the connected clients, with cooperative partition rebalancing +- Optional **poll auto-commit**, with processing and failure semantics explained in [concepts](/docs/introduction/concepts#polling-messages) +- **Consumer groups** distributing partitions across connected clients for horizontal scaling; ordering is per partition, with cooperative partition rebalancing - **Message expiry** with auto deletion based on the configurable **retention policy** - **Multi-tenant** support via abstraction of **streams** which group **topics** - **TLS** support for all transport protocols (TCP, QUIC, WebSocket, HTTPS) - **[Connectors](/docs/connectors/introduction)** - sinks, sources and data transformations based on the **custom Rust plugins** loaded dynamically at runtime -- **[Model Context Protocol](/docs/ai/mcp)** - provide context to LLM with **MCP server** exposing 40+ tools for full Iggy management +- **[Model Context Protocol](/docs/ai/mcp)** - provide context to LLM with **MCP server** exposing 40+ tools for streaming management - Optional server-side as well as client-side **data encryption** using AES-256-GCM - Optional metadata support in the form of **message headers** - Support for **OpenTelemetry** logs & traces + Prometheus metrics -- Built-in **[CLI](/docs/cli/start)** to manage the streaming server installable via `cargo install iggy-cli` +- Built-in **[CLI](/docs/cli/start)** to manage the streaming server installable via `cargo install iggy-cli --version 0.14.0-edge.7 --locked` - Built-in **[Web UI](/docs/web_ui/start)** dashboard (Svelte) that can be embedded directly in the server binary or run as a standalone container - Built-in **benchmarking app** to test the performance -- **Single binary deployment** (no external dependencies) -- **Accept-time connection distribution** across shards via file descriptor transfer for load balancing +- **Single binary deployment** without an external broker or database; operating-system libraries are still required by dynamically linked builds +- **Accept-time connection distribution** for plaintext TCP and WebSocket across shards via file descriptor transfer - Built on **Viewstamped Replication (VSR)** consensus: runs as a single node by default, with multi-node [clustering](/docs/clustering/vsr) available via the `[cluster]` configuration ## Supported languages SDK @@ -66,49 +68,49 @@ The name is an abbreviation for the Italian Greyhound - small yet extremely fast | Python | [apache-iggy](https://pypi.org/project/apache-iggy/) | PyPI | | Node.js | [apache-iggy](https://www.npmjs.com/package/apache-iggy) | npm | | Go | [iggy-go](https://pkg.go.dev/github.com/apache/iggy/foreign/go) | pkg.go.dev | -| PHP | [apache/iggy-php](https://packagist.org/packages/apache/iggy-php) | Packagist | +| PHP | [apache/iggy-php](https://github.com/apache/iggy/tree/master/foreign/php) | GitHub (source build) | | C++ | [iggy-cpp](https://github.com/apache/iggy/tree/master/foreign/cpp) | GitHub (WIP) | ## CLI -The interactive CLI is implemented under the `cli` project, to provide the best developer experience. This is a great addition to the Web UI, especially for developers who prefer using the console tools. +The interactive CLI is implemented under `core/cli`, to provide the best developer experience. This is a great addition to the Web UI, especially for developers who prefer using the console tools. -Iggy CLI can be installed with `cargo install iggy-cli` and then simply accessed by typing `iggy` in your terminal. It supports named connection contexts (profiles) for managing multiple server connections, shell completions for bash/zsh/fish/elvish/powershell, and session-based login on Linux. +Iggy CLI can be installed with `cargo install iggy-cli --version 0.14.0-edge.7 --locked` and then simply accessed by typing `iggy` in your terminal. It supports named connection contexts (profiles) for managing multiple server connections, shell completions for bash/zsh/fish/elvish/powershell, and session-based login through platform credential stores on Linux, macOS and Windows when the default `login-session` feature is enabled. ### Web UI The Web UI provides a comprehensive dashboard for the Iggy server, built with SvelteKit and TypeScript. It can run in two modes: -- **Embedded** - compiled into the server binary (with `iggy-web` feature flag), served at the `/ui` endpoint -- **Standalone** - as a separate container via `docker pull apache/iggy-web-ui` +- **Embedded** - compiled into the server binary (with `iggy-web` feature flag), served at the `/ui` endpoint when `http.web_ui = true` +- **Standalone** - as a separate container via `docker pull apache/iggy-web-ui:edge` -Features include stream/topic/partition management, message browser with JSON/string/XML decoders, user management, server logs viewer, real-time terminal, and server configuration overview. +Features include stream/topic/partition management, a message browser with JSON/string/XML decoders, and user management. The logs and terminal pages are placeholders, and the server settings page has a disabled Save action rather than an operational configuration editor. ## Connectors Iggy provides a highly performant and modular **[runtime](/docs/connectors/runtime)** for statically typed, yet dynamically loaded connectors. You can ingest data from external sources and push the data to Iggy streams, or fetch data from Iggy streams and forward it to external systems. **Create your own Rust plugins** by simply implementing either the `Source` or `Sink` trait and **build custom pipelines for the data processing**. -There are currently 14 sinks and 4 sources available. Sinks include PostgreSQL, MongoDB, Elasticsearch, ClickHouse, InfluxDB, Apache Iceberg, Delta Lake, Quickwit and S3. Sources include PostgreSQL, Elasticsearch, InfluxDB and random data generation. See the [connectors documentation](/docs/connectors/introduction) for the full catalog. +The source tree contains 16 sinks and 4 sources. Sinks include PostgreSQL, MongoDB, Elasticsearch, ClickHouse, InfluxDB, Apache Iceberg, Delta Lake, Quickwit and S3. Sources include PostgreSQL, Elasticsearch, InfluxDB and random data generation. See the [connectors documentation](/docs/connectors/introduction) for the full catalog. -The [docker image](https://hub.docker.com/r/apache/iggy-connect) is available, and can be fetched via `docker pull apache/iggy-connect`. +The [docker image](https://hub.docker.com/r/apache/iggy-connect) is available, and can be fetched via `docker pull apache/iggy-connect:edge`. ## Model Context Protocol The [Model Context Protocol](https://modelcontextprotocol.io) (MCP) is an open protocol that standardizes how applications provide context to LLMs. The **[Iggy MCP Server](/docs/ai/mcp)** is an implementation of the MCP protocol for message streaming infrastructure. It exposes 40+ tools covering streams, topics, partitions, messages, consumer groups, users, and more. It supports both HTTP and stdio transports, making it compatible with tools like Claude Desktop and other MCP clients. -The [docker image](https://hub.docker.com/r/apache/iggy-mcp) is available, and can be fetched via `docker pull apache/iggy-mcp`. +The [docker image](https://hub.docker.com/r/apache/iggy-mcp) is available, and can be fetched via `docker pull apache/iggy-mcp:edge`. ### Docker -The official Apache Iggy images can be found on [Docker Hub](https://hub.docker.com/r/apache/iggy). Simply type `docker pull apache/iggy` to pull the image. +The official Apache Iggy images can be found on [Docker Hub](https://hub.docker.com/r/apache/iggy). These docs prepare for server 0.9.0. Use `docker pull apache/iggy:0.9.0` when that release is available, or `docker pull apache/iggy:edge` for a development image. SDK and tooling versions are independent of the server version; the CLI command above pins a published edge version. You can also find the images for all the different tooling such as Connectors, MCP Server etc. [here](https://hub.docker.com/u/apache?page=1&search=iggy). Please note that the images tagged as `latest` are based on the official, stable releases, while the `edge` ones are updated directly from latest version of the `master` branch. -You can find the `Dockerfile` and `docker-compose` in the root of the repository. To build and start the server, run: `docker compose up`. +You can find the development `Dockerfile` and `docker-compose.yml` in the root of the repository. Published server images use `core/server/Dockerfile`. To build and start the server, run: `docker compose up`. -Additionally, you can run the `CLI` which is available in the running container, by executing: `docker exec -it iggy-server /iggy`. +For the root Compose build, run the included CLI with `docker exec -it iggy-server /iggy`. In published images, its path is `/usr/local/bin/iggy`. Keep in mind that running the container on operating systems other than Linux, where the Docker is running in the VM, might result in the performance degradation. @@ -128,12 +130,12 @@ ulimits: Or when running with `docker run`: ``` -docker run --cap-add=SYS_NICE --security-opt seccomp=unconfined --ulimit memlock=-1:-1 apache/iggy:latest +docker run --cap-add=SYS_NICE --security-opt seccomp=unconfined --ulimit memlock=-1:-1 -p 8090:8090 -e IGGY_TCP_ADDRESS=0.0.0.0:8090 -e IGGY_NODE_ADVERTISED_ADDRESS=localhost apache/iggy:edge ``` ### Helm Charts -Helm charts for Kubernetes deployment are available in the [repository](https://github.com/apache/iggy/tree/master/helm/charts/iggy). The chart includes templates for Deployment, Service, ServiceAccount, HPA (Horizontal Pod Autoscaler), Ingress, PersistentVolumeClaim, ServiceMonitor (for Prometheus), and root user credentials Secret. +Helm charts for Kubernetes deployment are available in the [repository](https://github.com/apache/iggy/tree/master/helm/charts/iggy). The chart includes templates for Deployment, Service, ServiceAccount, Ingress, PersistentVolumeClaim, ServiceMonitor (for Prometheus), and root user credentials Secret. ### Versioning diff --git a/content/docs/introduction/architecture.mdx b/content/docs/introduction/architecture.mdx index f6032bcab7..5968a61898 100644 --- a/content/docs/introduction/architecture.mdx +++ b/content/docs/introduction/architecture.mdx @@ -7,7 +7,7 @@ This page covers the internals of the Iggy server: how work is scheduled across ## How a message flows through Iggy -Before diving into the architecture details, here's the complete journey of a message from client to disk: +Before diving into the architecture details, here's an overview of a message's journey from client to disk: @@ -19,20 +19,20 @@ Iggy uses a **thread-per-core shared nothing architecture** combined with `io_ur ### How it works -Each CPU core runs its own **shard** (an instance of `IggyShard`), pinned to a specific core via `sched_setaffinity` on Linux. Each shard has its own single-threaded `compio` async runtime, which means there is no cross-thread synchronization needed within a shard. Memory is bound to the NUMA node of the core via `hwlocality` for optimal memory access latency. +Each configured **shard** (an instance of `IggyShard`) has its own single-threaded `compio` async runtime. With `pin_cores = true`, Linux shard threads are pinned to their selected CPUs via `sched_setaffinity`. NUMA allocation modes also bind memory via `hwlocality`. Setting `pin_cores = false` disables both bindings; shared metadata and inter-shard communication still require synchronization. ### Shard roles and connection distribution Shard 0 has a special role: it binds **every listener** - the replica plane and all client transports (TCP, QUIC, WebSocket, HTTP). Connections are then spread across shards at accept time: - **Plaintext TCP and WebSocket** connections are handed off round-robin to peer shards. Shard 0's coordinator duplicates the socket's file descriptor, ships a connection-setup frame to the target shard, and drops its own handle, so the owning shard serves the connection from then on - before a single byte is read. -- **QUIC and TLS-wrapped TCP** connections terminate on shard 0, because their per-connection state cannot be moved between shards. +- **QUIC, TLS-wrapped TCP and secure WebSocket (WSS)** connections terminate on shard 0, because their per-connection state cannot be moved between shards. HTTP is also served on shard 0. All shards, **including shard 0**, own partitions and serve partition requests. ### Request routing -Requests are routed between shards using **message passing** (via `crossfire` bounded mpsc channels), which avoids locking entirely. The routing logic splits operations into two planes: +Requests are routed between shards using **message passing** (via `crossfire` bounded mpsc channels), so partition state remains on its owning shard. The routing logic splits operations into two planes: - **Metadata operations** (create/delete stream/topic/user etc.) always execute on **shard 0** - it is the only shard that commits metadata - **Partition operations** (send_messages, poll_messages, store_consumer_offset) are routed to the shard owning that partition via a lock-free concurrent map lookup. A request that lands on a non-owning shard rides the inter-shard message bus to the owner @@ -50,27 +50,29 @@ The `IggyNamespace` packs stream_id (20 bits), topic_id (12 bits), and partition The sharding system supports multiple allocation modes via the `cpu_allocation` config: - `"all"` - one shard per available CPU core -- A numeric value (e.g. `4`) - exactly N shards pinned to cores 0..N -- A range (e.g. `"5..8"`) - shards on specified core range -- `"numa:auto"` - automatically detect NUMA topology and bind accordingly +- A numeric value (e.g. `4`) - exactly N shards, pinned to the first N CPUs in the process's allowed CPU set when pinning is enabled +- A range (e.g. `"5..8"`) - shards on CPUs 5, 6 and 7 when pinning is enabled; those CPUs must be allowed for the process +- `"numa:auto"` - automatically detect NUMA topology and select physical cores, avoiding sibling hyperthreads - `"numa:nodes=0,1;cores=4;no_ht=true"` - fine-grained NUMA control per node with hyperthread avoidance ### io_uring and compio -Traditional async runtimes like tokio use `epoll` which is **readiness-based** - you ask the kernel "is this file descriptor ready?" and then perform the I/O yourself. The Linux kernel considers regular files "always ready" for epoll, which means tokio has to outsource file I/O to a blocking thread pool (up to 512 threads). This does not scale well. +Traditional async runtimes like tokio use `epoll` which is **readiness-based** - you ask the kernel "is this file descriptor ready?" and then perform the I/O yourself. [Regular files cannot be registered with epoll](https://man7.org/linux/man-pages/man2/epoll_ctl.2.html). Tokio runs file I/O on a blocking thread pool (512 threads by default, configurable). This does not scale well. -`io_uring` is **completion-based** - you submit I/O requests to a submission queue (SQ), and the kernel completes them and places results in a completion queue (CQ). Both queues are lock-free ring buffers shared between user space and kernel. This is fundamentally better for disk I/O. +`io_uring` is **completion-based** - you submit I/O requests to a submission queue (SQ), and the kernel completes them and places results in a completion queue (CQ). Both queues are shared ring buffers between user space and kernel. Submissions and completions can be batched to reduce syscalls. This is fundamentally better for disk I/O. Iggy uses **compio** as its async runtime, which provides a driver-disaggregated architecture on top of io_uring (Linux) and IOCP (Windows). Each shard gets its own compio executor configured with: - Capacity: 4096 concurrent I/O operations (by default) -- Event interval: 128 events per loop iteration +- Event interval: poll the I/O driver after 128 scheduler ticks (roughly task polls) by default - Cooperative task running enabled +`IGGY_SHARD_RUNTIME_CAPACITY` and `IGGY_SHARD_EVENT_INTERVAL` override the two numeric defaults. On macOS, compio uses its polling driver and a blocking pool for file I/O. + ### Performance: Tokio vs Thread-per-Core -The migration from Tokio to thread-per-core with compio delivered significant latency improvements across the board: +The [historical migration benchmarks](https://iggy.apache.org/blogs/2026/02/27/thread-per-core-io_uring/) compared v0.5.0 with v0.7.0 at approximately 1,000 MB/s per node. The chart shows selected latency reductions for 8, 16 and 32 producers, each using its own stream. The 8-producer case also reported higher P95 and P99 latency; these are not measurements of 0.9.0: @@ -97,6 +99,7 @@ local_data/ ├── logs/ │ └── iggy-server.log ├── state/ +│ └── log/ └── streams/ └── 0/ └── topics/ @@ -105,14 +108,18 @@ local_data/ └── 0/ ├── 00000000000000000000.index ├── 00000000000000000000.log + ├── superblock.a + ├── superblock.b └── offsets/ + ├── consumers/ + └── groups/ ``` -The stream, topic and partition directories are named after their numeric IDs, **assigned from 0**. The `metadata/journal.wal` file is the VSR write-ahead log that persists all metadata operations (stream/topic/user creation, etc.). The `runtime/current_config.toml` file captures the configuration the server actually booted with. Segment files are named by the 20-digit start offset of their first record. The `.index` file is created automatically and speeds up searches by keeping track of the offsets and timestamps of the records. The `offsets/` directory holds the server-side consumer and consumer group offsets. +This example shows a fresh partition with the default paths. The stream, topic and partition directories are named after their numeric IDs, **assigned from 0**. The `metadata/journal.wal` file is the VSR write-ahead log that persists all metadata operations (stream/topic/user creation, etc.). The `runtime/current_config.toml` file captures the configuration the server actually booted with. Segment files are named by their 20-digit start offset. Recovery can create an empty active segment at a reserved offset beyond the last stored message; partition superblocks record the recovery frontiers. The `.index` file is created automatically and speeds up searches by keeping track of the offsets and timestamps of the records. The `offsets/` directory holds the server-side consumer and consumer group offsets. ## Memory pool -Iggy uses a custom memory pool with 28 buckets holding buffer sizes from 4 KiB to 512 MiB. The default pool size is 4 GiB with up to 8192 buffers per bucket. This eliminates allocation overhead on the hot path and enables zero-copy message passing between components. The pool is page-aligned (4096-byte multiples) and requires a **minimum of 512 MiB**. +Iggy uses a custom memory pool with 28 buckets holding buffer sizes from 4 KiB to 512 MiB. The default pool size is 4 GiB with up to 8192 buffers per bucket. Buffers are allocated on demand and reused; requests that exceed the pool budget can allocate outside it. Frozen buffers can be shared between components without copying their contents. The pool is page-aligned (4096-byte multiples) and requires a **minimum of 512 MiB**. ## Write pipeline @@ -120,8 +127,8 @@ Messages flow through a multi-stage write pipeline: 1. Messages arrive on the owning shard and are buffered in the partition journal. 2. Partition VSR replicates prepares. With `durability=persisted`, a multi-replica group requires recoverable prepare-WAL copies at the replication quorum before commit. -3. Committed operations are applied before success is returned. A singleton with `durability=persisted` synchronizes local segment state before replying. -4. Ordinary segment writes use per-topic count and byte thresholds (defaults: 1024 messages, 1 MiB); required persistence, capacity pressure, and lifecycle work can flush earlier. The `MessagesWriter` uses **vectored I/O** with up to 1024 buffers per syscall. +3. Awaited writes apply committed operations before success is returned. A singleton with `durability=persisted` synchronizes local segment state before replying. +4. Ordinary segment writes use per-topic count and byte thresholds (defaults: 1024 messages, 1 MiB); required persistence, capacity pressure, and lifecycle work can flush earlier. The `MessagesWriter` uses **vectored I/O** in chunks of up to 1024 buffers. Partial writes can require more than one I/O submission. 5. When a segment reaches the topic's segment size (default 1 GiB), it is **sealed** and a new segment is created. Message `durability` and `consumer_offset_durability` default independently to `replicated`. Both policies write data to disk; `persisted` adds a stable-storage requirement at completion. See [Durability](/docs/server/durability). diff --git a/content/docs/introduction/concepts.mdx b/content/docs/introduction/concepts.mdx index e1157cd1a7..101cfff1b9 100644 --- a/content/docs/introduction/concepts.mdx +++ b/content/docs/introduction/concepts.mdx @@ -3,15 +3,15 @@ title: Concepts description: "The domain model behind Iggy, and how an append-only streaming log differs from a message broker." --- -Iggy is a persistent message streaming platform: messages are stored in a form of an **append-only log**. You can create multiple streams, consisting of topics, which might have one or more partitions assigned, e.g. to achieve the horizontal scalability between many independent consumers or higher system resiliency. You can think of Iggy as an alternative to Kafka or RabbitMQ streams. +Iggy is a persistent message streaming platform: messages are stored in a form of an **append-only log**. You can create multiple streams, consisting of topics, which might have one or more partitions assigned, e.g. to divide consumption among independent consumers. Multi-node replication provides redundancy. You can think of Iggy as an alternative to Kafka or RabbitMQ streams. ## Message streaming You've probably used RabbitMQ or Kafka already. They look similar at the first glance, and you can achieve the similar results with both (e.g. publishing and consuming the events by the different applications built on top of microservices architecture), but they work differently underneath. -The main difference is that RabbitMQ (except the recently released Streams plugin) is the **message broker**, which means that it's responsible for delivering the messages to the consumers. It works in the FIFO (First In, First Out) manner and the messages are being kept in the queues. For example, if you have multiple, distinct consumers, then each one would create its own queue, the message would be replicated between each queue and each consumer would be responsible for reading the messages from its own queue. Once the message is processed, it's gone from the queue, so there's **no built-in way to replay** past the messages. The more consumers you have, the more queues you have to create, which might result in more resources being used. The typical message broker follows the so-called smart pipes and dumb endpoints pattern. +RabbitMQ is a **message broker** with both queues and [streams](https://www.rabbitmq.com/docs/streams). Traditional queues deliver messages to consumers and remove them after acknowledgement, or on delivery when automatic acknowledgement is used, so acknowledged messages cannot be replayed from the queue. Multiple consumers can share a queue and divide its messages. Independent subscribers that each need a copy use separate queues bound to an exchange. Queues normally use FIFO (First In, First Out) order, but priorities, redelivery and competing consumers can affect the observed order. See [RabbitMQ's queue semantics](https://www.rabbitmq.com/docs/queues). The typical message broker follows the so-called smart pipes and dumb endpoints pattern. -On the other hand, Kafka is a **message streaming platform**, meaning that it's not responsible for delivering the messages to the consumers, but rather it's storing them in a form of an append-only log. The consumers are responsible for reading the messages from the log and processing them. You might have multiple distinct consumers, and it doesn't affect the resource usage as there's only one log. The consumers can read the messages from the beginning, or from the specific offset, thus you can replay the messages. The typical message streaming platform follows the so-called dumb pipes and smart endpoints pattern. +On the other hand, Kafka is a **message streaming platform** that stores messages in an append-only log and serves [consumer fetch requests](https://kafka.apache.org/41/design/design/). The consumers are responsible for reading the messages from the log and processing them. Multiple consumer groups can read the same retained log without a separate stored copy for each group, although their fetch requests still consume CPU and network resources. The consumers can read the messages from the beginning, or from the specific offset, thus you can replay the messages. The typical message streaming platform follows the so-called dumb pipes and smart endpoints pattern. Both approaches have advantages and disadvantages. The message broker is a more mature concept, but the message streaming platform is gaining more and more popularity, especially in the cloud-native world. And you can achieve much higher performance and throughput with the message streaming platform, since it acts as a simple database, being optimized for the append-only operations and can be queried in a very efficient way. @@ -19,9 +19,9 @@ Iggy is the latter, a message streaming platform. ## Append-only log -The append-only log is the core concept of Iggy. It's a simple data structure, which is optimized for the append-only operations. It's a sequence of records, that are being appended to the end of the log. The records are **immutable**, so that they can't be changed once they are written to the log. The records are being written in the order they are received, which results in the log being ordered. +The append-only log is the core concept of Iggy. It's a simple data structure, which is optimized for the append-only operations. It's a sequence of records, that are being appended to the end of the log. The records are **immutable**, so that they can't be changed once they are written to the log. Records within a partition follow the order admitted by that partition's primary. There is no global ordering across partitions. -You address the log by **offset**, the position of the record in the log. The offset is a simple integer that starts from 0 and is incremented by 1 for each record. When the client reads the records, it specifies the offset to start from and the maximum number of records it wants. Starting from the beginning, or from any earlier offset, is how you replay the messages. +You address the log by **offset**, the position of the record in the log. Offsets start from 0 and increase within a partition. They are not guaranteed to be contiguous: after recovery, the server can skip reserved offsets to avoid reusing them. When the client reads the records, it specifies the offset to start from and the maximum number of records it wants. Starting from the beginning, or from any earlier offset, is how you replay the messages. @@ -30,27 +30,27 @@ You address the log by **offset**, the position of the record in the log. The of ## Stream While we could put an equal sign between the log and the stream, they are not the same, at least in a case of Iggy streaming server. -The stream is a logical concept, and you might think of it as a **namespace**. For example, you could have a single stream for the whole system, or multiple streams e.g. representing the different environments, such as `dev`, `staging` and `production`. The stream is identified by its unique ID. The stream can have one or more topics assigned, which results in the records being published to the specific topics that belong to the particular stream. +The stream is a logical concept, and you might think of it as a **namespace**. For example, you could have a single stream for the whole system, or multiple streams e.g. representing the different environments, such as `dev`, `staging` and `production`. The stream is identified by its unique ID. The stream can have zero or more topics assigned, which results in the records being published to the specific topics that belong to the particular stream. ## Topic -The topic is also the logical concept, which is a part of the stream. The topic is identified by its unique ID. You could think of topic as an entity being responsible for storing the specific type of the records. For example, you could have a topic for the user events, and another topic for the order events, etc. +The topic is also the logical concept, which is a part of the stream. The topic is identified by its ID, which is unique within its stream. You could think of topic as an entity being responsible for storing the specific type of the records. For example, you could have a topic for the user events, and another topic for the order events, etc. -The messages are not being stored in the topic directly, but rather in the **partitions**, which are assigned to the topic. The topic can have one or more partitions assigned, that could help achieve higher parallelism and throughput. The topic can also have the **retention policy** assigned, which means that the records are being deleted automatically once they are older than the specified retention period. Topics also support maximum size limits and per-topic storage options (`segment_size`, `durability`, `consumer_offset_durability`, and flush thresholds) set at creation, plus a `compression_algorithm` option (a placeholder today: no compression is applied yet). Both [durability policies](/docs/server/durability) independently default to `replicated`. +The messages are not being stored in the topic directly, but rather in the **partitions**, which are assigned to the topic. The topic can have one or more partitions assigned, that could help achieve higher parallelism and throughput. The topic can also have a **retention policy**. Expiry removes whole sealed segments after their newest message has expired, subject to the stored consumer-offset barrier; it does not delete each message immediately at its expiry time. See [topic options](/docs/server/topic-options) for expiry and size-limit behavior. Topics also support maximum size limits and per-topic storage options (`segment_size`, `durability`, `consumer_offset_durability`, and flush thresholds) set at creation, plus a `compression_algorithm` option (a placeholder today: no compression is applied yet). Both [durability policies](/docs/server/durability) independently default to `replicated`. ## Partition -The partition has its own unique ID and belongs to the topic. The partition is responsible for storing the records. The records are being distributed between the partitions, therefore the partition acts as a simple database, which is optimized for the append-only operations. The partition is identified by its unique ID, which is an integer. Stream, topic and partition IDs are all assigned **starting from 0**, incremented by 1 for each new one. The partition ID is unique per topic, thus the same partition ID can be used in multiple topics. +The partition has its own unique ID and belongs to the topic. The partition is responsible for storing the records. The records are being distributed between the partitions, therefore the partition acts as a simple database, which is optimized for the append-only operations. The partition is identified by its unique ID, which is an integer. Stream, topic and partition IDs are assigned **starting from 0**. Deleted stream and topic IDs can be reused. New partitions use IDs above the highest remaining partition ID, so deleting the highest partitions can also allow their IDs to be reused. The partition ID is unique per topic, thus the same partition ID can be used in multiple topics. Thanks to having multiple partitions, we can achieve the horizontal scalability between many independent consumers, since each consumer can read the messages from the different partitions. This can be achieved by using more advanced concepts such as consumer groups. -Each partition in the thread-per-core architecture is owned by **exactly one shard** and includes: +Each partition in the thread-per-core architecture is owned by **exactly one shard per replica** and includes: - A `SegmentedLog` with sealed segments and one active segment - Consumer offsets and consumer group offsets ## Segment -The segment, being a part of the partition, is the actual **physical layer** which stores the records in the binary format in a form of the files. Each segment has the limited size (1 GiB unless the topic sets its own `segment_size`) and once it's full, the new segment is being created automatically. The segment name is based on the start offset of the first record in the segment and is unique per partition. +The segment, being a part of the partition, is the actual **physical layer** which stores the records in the binary format in a form of the files. Each segment has a soft size limit (1 GiB unless the topic sets its own `segment_size`). A segment can exceed it by one whole batch before it is sealed and a new segment is created. The segment name is based on the start offset of the first record in the segment and is unique per partition. Each segment consists of: - `.log` file - the actual message data @@ -63,11 +63,11 @@ Consumers can poll the messages in multiple ways: - **By offset** - start reading from the specified offset. The client tracks its own position. - **By timestamp** - start reading from the first message at or after the given timestamp. - **First / Last** - start from the beginning or the end of the partition. -- **Next** - continue from the consumer offset stored on the server side. The client no longer needs to track the offset itself. Combine it with `auto_commit: true` to commit the offset automatically once the messages are fetched (*at-most-once* delivery), or call `store_offset()` explicitly after processing. +- **Next** - continue from the consumer offset stored on the server side. The server tracks the cursor. On the primary, `auto_commit: true` submits an offset update before the poll response is delivered, without waiting for that update to commit. Follower polls do not advance it. This alone guarantees neither at-most-once nor at-least-once processing. For processing-before-commit ordering, explicitly store the offset after processing, using `store_consumer_offset()` in the Rust SDK. ## Consumer groups -Consumer groups provide horizontal scaling for message consumption. When multiple consumers join the same consumer group, the server automatically distributes partitions among group members so that each partition is consumed by **exactly one member**. When members join or leave, the server triggers a **cooperative partition rebalancing** with a pending revocation phase (configurable timeout, default 30s) to ensure smooth transitions without message loss. +Consumer groups provide horizontal scaling for message consumption. When multiple consumers join the same consumer group, the server automatically distributes partitions among group members so that each partition has **at most one member permitted to poll it** within that group. When members join or leave, the server triggers a **cooperative partition rebalancing** with a pending revocation phase (configurable timeout, default 30s) to let the previous owner finish processing and commit before handoff. Application processing and offset-commit ordering still determine whether failures cause skips or duplicates. @@ -75,19 +75,19 @@ Consumer groups provide horizontal scaling for message consumption. When multipl -In the SDKs, each message carries a 64-byte in-memory header (little-endian fields). On the wire and on disk, messages travel inside batch records with a compact 48-byte per-message frame. See the [Binary Protocol](/docs/binary-protocol) section for the exact encodings. +The Rust SDK's `IggyMessageHeader::to_bytes()` representation is 64 bytes, with little-endian fields. The diagram and table describe that SDK representation, not Rust struct memory layout. On the wire and on disk, messages travel inside batch records with a compact 48-byte per-message frame header. See the [Binary Protocol](/docs/binary-protocol) section for the exact encodings. | Field | Bytes | Type | Description | |-------|-------|------|-------------| | checksum | 0-8 | u64 | xxHash3 integrity checksum | -| id | 8-24 | u128 | Unique message ID (UUIDv4) | -| offset | 24-32 | u64 | Sequential offset in partition | +| id | 8-24 | u128 | Client-supplied 128-bit ID; generated as UUIDv4 when omitted by the Rust SDK | +| offset | 24-32 | u64 | Increasing offset within the partition | | timestamp | 32-40 | u64 | Server-assigned timestamp | | origin_timestamp | 40-48 | u64 | Client-provided timestamp | | user_headers_length | 48-52 | u32 | Length of optional headers | | payload_length | 52-56 | u32 | Length of payload | | reserved | 56-64 | u64 | Reserved (must be 0) | -After the header comes the optional user headers bytes, followed by the payload bytes. +In the SDK message representation, the header is followed by payload bytes and then optional user-header bytes. To see how the server schedules these concepts across CPU cores and stores them on disk, head over to [architecture](/docs/introduction/architecture). diff --git a/content/docs/introduction/getting-started.mdx b/content/docs/introduction/getting-started.mdx index 1f5bc12fbc..00f87c1708 100644 --- a/content/docs/introduction/getting-started.mdx +++ b/content/docs/introduction/getting-started.mdx @@ -21,7 +21,7 @@ The completed sample can be found in the [repository](https://github.com/apache/ For our purpose, we will focus on the basic scenario in order to keep things simple. Before we begin implementing the consumer and producer apps, we need to start the Iggy streaming server. -The quickest way is Docker, using the [official images](https://hub.docker.com/r/apache/iggy): +This guide targets server **0.9.0**. Once that release is published, you can use its [official Docker image](https://hub.docker.com/r/apache/iggy). During release preparation, use a matching `edge` build or the source build below: ```bash docker run --rm \ @@ -30,7 +30,7 @@ docker run --rm \ -e IGGY_TCP_ADDRESS=0.0.0.0:8090 \ -e IGGY_NODE_ADVERTISED_ADDRESS=localhost \ -e IGGY_ROOT_USERNAME=iggy -e IGGY_ROOT_PASSWORD=iggy \ - apache/iggy:latest + apache/iggy:0.9.0 ``` The capabilities, seccomp setting, and memlock limit form a permissive development setup. Production deployments can use narrower syscall permissions and a finite memory budget; see [Docker & Helm](/docs/server/docker#why-these-capabilities) for the details. `IGGY_TCP_ADDRESS` is needed because the server binds to `127.0.0.1` inside the container by default, which a published port cannot reach. `IGGY_NODE_ADVERTISED_ADDRESS` is needed because that wildcard leaves the server with no address to give clients, and it refuses to start rather than publish one nobody can dial. Here the port is published to the host, so `localhost` is that address. Setting the root credentials explicitly means the username and password used later in this guide will work. @@ -41,7 +41,7 @@ Alternatively, build from source by cloning the [repository](https://github.com/ cargo run --bin iggy-server -- --fresh --with-default-root-credentials ``` -A bare `iggy-server` boot does **not** create the root user with the well-known `iggy`/`iggy` pair. It generates a random password and prints it **exactly once** to the logs. The `--with-default-root-credentials` flag switches to the development credentials (username: `iggy`, password: `iggy`), and `--fresh` wipes the data directory first, so the root user is created anew with them. You can also set `IGGY_ROOT_USERNAME` and `IGGY_ROOT_PASSWORD` yourself. The environment **takes precedence** over the flag, and **only the first boot** on an empty data directory reads these values. +A bare `iggy-server` boot does **not** create the root user with the well-known `iggy`/`iggy` pair. It generates a random password and prints it **exactly once** to the logs. The `--with-default-root-credentials` flag switches to the development credentials (username: `iggy`, password: `iggy`), and `--fresh` wipes the data directory first, so the root user is created anew with them. You can also set `IGGY_ROOT_USERNAME` and `IGGY_ROOT_PASSWORD` yourself. The environment **takes precedence** over the flag. The credentials initialize the stored root account **only on first creation**; changing them on a later boot does not reset its password. Supplied environment credentials are still validated on restarts. All the data used by the server will be persisted under the `local_data` directory, unless specified differently in the configuration. @@ -95,7 +95,7 @@ From that point on, we will focus on implementing the message streaming between ## Building the producer -We will begin with installing the Iggy client crate - execute `cargo add iggy` in your terminal. Next, install [tokio.rs](https://tokio.rs) dependency with `cargo add tokio` as we will use the asynchronous runtime. Eventually, modify your `main.rs`, so it looks like this: +We will begin with installing the Iggy client crate - use the SDK from the same source checkout as your server. With `iggy-sample` beside the `iggy` repository, execute `cargo add iggy --path ../iggy/core/sdk` in your terminal. Published SDK releases can expose different APIs. Next, install [tokio.rs](https://tokio.rs) dependency with `cargo add tokio --features macros,rt-multi-thread,time` as we will use the asynchronous runtime. Eventually, modify your `main.rs`, so it looks like this: ```rust use std::error::Error; @@ -113,9 +113,9 @@ let client = IggyClient::default(); client.connect().await?; ``` -The default server address being `127.0.0.1:8090` is configured on the server side, and can be easily adjusted by updating `config.toml` in the `core/server` directory. +Both the default TCP client and the server use `127.0.0.1:8090`. If you change the server address in `core/server/config.toml`, configure the client to connect to that address too. -We could make use of more advanced components such as [`ClientProvider`](https://github.com/apache/iggy/blob/master/core/sdk/src/client_provider.rs), pass the custom configuration built via console args to choose between the different protocols as all the available clients implement the same [Client](https://github.com/apache/iggy/blob/master/core/sdk/src/clients/client.rs) trait and so on. +We could make use of more advanced components such as [`ClientProvider`](https://github.com/apache/iggy/blob/master/core/sdk/src/client_provider.rs), pass the custom configuration built via console args to choose between the different protocols as all the available clients implement the same [Client](https://github.com/apache/iggy/blob/master/core/common/src/traits/client.rs) trait and so on. If you're eager to find out how to build more advanced (and configurable) applications, check the Rust [examples](/docs/sdk/rust/examples). Nevertheless, let's focus on implementing our producer side :) @@ -129,9 +129,9 @@ client When you start the application now, by running `cargo r --bin producer` it will execute immediately, however, you should be able to see the logs on Iggy server - connection should be accepted, user logged in, and then connection should be closed right away, meaning that we've just established our very first communication with the streaming server. -Before we will move on with the code, let's include some logging in our apps as well (Iggy client already has some logging in place on different levels). Let's make use of the [tracing.rs](https://tracing.rs) crates: `cargo add tracing tracing_subscriber`. +Before we will move on with the code, let's include some logging in our apps as well (Iggy client already has some logging in place on different levels). Let's make use of the [tracing.rs](https://tracing.rs) crates: `cargo add tracing` and `cargo add tracing-subscriber --features env-filter`. -To make use of logging, simply invoke `tracing_subscriber::fmt::init()` at the beginning of `main()` method: +Initialize the tracing subscriber at the beginning of `main()`, using `RUST_LOG` when set and `info` otherwise: ```rust use std::error::Error; @@ -139,7 +139,12 @@ use iggy::prelude::*; #[tokio::main] async fn main() -> Result<(), Box> { - tracing_subscriber::fmt::init(); + tracing_subscriber::fmt() + .with_env_filter( + tracing_subscriber::EnvFilter::try_from_default_env() + .unwrap_or_else(|_| "info".into()), + ) + .init(); let client = IggyClient::default(); client.connect().await?; client @@ -153,9 +158,9 @@ From that point on, when starting the application, you should be able to see at So far, so good, however, before we will be able to publish any messages to our streaming server, at first, we need to create the stream, topic and partition(s) - if you're unfamiliar with these concepts, please refer to [concepts](/docs/introduction/concepts) where all of them are described in-depth. -Since our `IggyClient` implements the common [Client](https://github.com/apache/iggy/blob/master/core/sdk/src/clients/client.rs) trait, you can find lots of the different methods to interact with the server, also from the administrative point of view, e.g. creating the streams, topics etc. +Since our `IggyClient` implements the common [Client](https://github.com/apache/iggy/blob/master/core/common/src/traits/client.rs) trait, you can find lots of the different methods to interact with the server, also from the administrative point of view, e.g. creating the streams, topics etc. -These methods are **not idempotent** - for example, if you were to try creating the stream with the same name which already exists on the server, you would receive the specific error. In such a case, you can simply check for an error and move on. When creating a stream, we only provide its name - the server assigns the numeric ID automatically. Stream, topic and partition IDs are all assigned **starting from 0**. Let's do this then :) +These methods are **not idempotent** - for example, if you were to try creating the stream with the same name which already exists on the server, you would receive the specific error. In such a case, handle the specific already-exists error and propagate other failures. When creating a stream, we only provide its name - the server assigns the numeric ID automatically. Stream, topic and partition IDs are all assigned **starting from 0**. Let's do this then :) ```rust use iggy::prelude::*; @@ -167,20 +172,28 @@ const TOPIC_NAME: &str = "sample-topic"; #[tokio::main] async fn main() -> Result<(), Box> { - tracing_subscriber::fmt::init(); + tracing_subscriber::fmt() + .with_env_filter( + tracing_subscriber::EnvFilter::try_from_default_env() + .unwrap_or_else(|_| "info".into()), + ) + .init(); let client = IggyClient::default(); client.connect().await?; client .login_user(DEFAULT_ROOT_USERNAME, DEFAULT_ROOT_PASSWORD) .await?; - init_system(&client).await; + init_system(&client).await?; Ok(()) } -async fn init_system(client: &IggyClient) { +async fn init_system(client: &IggyClient) -> Result<(), IggyError> { match client.create_stream(STREAM_NAME).await { Ok(_) => info!("Stream was created."), - Err(_) => warn!("Stream already exists and will not be created again."), + Err(IggyError::StreamNameAlreadyExists(_)) => { + warn!("Stream already exists and will not be created again."); + } + Err(error) => return Err(error), } match client @@ -196,8 +209,12 @@ async fn init_system(client: &IggyClient) { .await { Ok(_) => info!("Topic was created."), - Err(_) => warn!("Topic already exists and will not be created again."), + Err(IggyError::TopicNameAlreadyExists(..)) => { + warn!("Topic already exists and will not be created again."); + } + Err(error) => return Err(error), } + Ok(()) } ``` @@ -240,7 +257,7 @@ pub struct Identifier { } ``` -Whenever we interact with the streaming server in terms of e.g. sending or polling the messages, managing the streams, topics etc. we need to provide the unique identifier of the stream and the topic that we want to use. Since each stream and topic have a unique numeric ID, as well as a unique name, we can use either of them. Here, we use the string name by calling `Identifier::named()` (implicitly via `TryFrom` on a `&str`), however, you can also use the numeric identifier by invoking the `Identifier::numeric()` method instead. It's up to your preference if you'd rather work with the identifier being a number or string when building your applications. +Whenever we interact with the streaming server in terms of e.g. sending or polling the messages, managing the streams, topics etc. we need to provide the unique identifier of the stream and the topic that we want to use. Since each stream and topic have a unique numeric ID, as well as a unique name, we can use either of them. Use `Identifier::named()` to force a name, or `Identifier::numeric()` for a numeric ID. `TryFrom<&str>` interprets a string that parses as a `u32` as a numeric ID; other strings, including the names in this sample, become named identifiers. It's up to your preference if you'd rather work with the identifier being a number or string when building your applications. Next, let's move onto the `partitioning` field: @@ -257,7 +274,7 @@ In our scenario, we simply make use of `PartitioningKind::PartitionId` (by invok However, once your system grows, you might want to parallelize the messages across the independent consumers, in order to achieve the horizontal scaling, higher resiliency etc. In that case, you might consider using either `PartitioningKind::Balanced` (the SDK picks the next partition using a client-local round-robin e.g. 0->1->2->0->1->2 etc.) or `PartitioningKind::MessagesKey` instead (e.g. by invoking one of the helper methods `messages_key()`), where the value wouldn't be a partition ID anymore (the SDK hashes the provided value with xxHash32 modulo the partition count), but, as the name states, some kind of identifier, which is unique for all the messages that should have **guaranteed ordering**. The HTTP API instead sends the chosen strategy to the server, which resolves it at admission. -For example, given that you process the set of messages related to the specific order ID (e.g. created, confirmed, paid, delivered etc.), you could use that as a key value to ensure that all the messages which are part of the specific workflow, will always be put onto the same partition. The value of the key can be anything (e.g. string, number) with the **maximum length of 255 bytes**. +For example, given that you process the set of messages related to the specific order ID (e.g. created, confirmed, paid, delivered etc.), you could use that as a key value to ensure that all the messages which are part of the specific workflow, will be put onto the same partition while the partition count and partitioning strategy stay unchanged. The value of the key can be anything (e.g. string, number) with the **maximum length of 255 bytes**. Anyway, let's get back to our scenario, and consider the following code responsible for publishing the messages: @@ -297,7 +314,7 @@ async fn produce_messages(client: &IggyClient) -> Result<(), Box> { } ``` -The reason behind passing a mutable reference is that the underlying client (especially when using the `IggyClient` wrapper on top of the low-level client) might want to modify the command before sending it to the server. For example, the client might want to encrypt the payload, include the default headers, or provide a custom `Partitioner` implementation etc. thus by using `&mut` we can avoid copying the command each time. +The mutable slice lets `IggyClient` encrypt message payloads and user headers in place when client-side encryption is enabled. Finally, let's complete the implementation of the producer - once you start the application after the latest changes, you shall see the messages being sent to the newly created stream. @@ -315,21 +332,29 @@ const PARTITION_ID: u32 = 0; #[tokio::main] async fn main() -> Result<(), Box> { - tracing_subscriber::fmt::init(); + tracing_subscriber::fmt() + .with_env_filter( + tracing_subscriber::EnvFilter::try_from_default_env() + .unwrap_or_else(|_| "info".into()), + ) + .init(); let client = IggyClient::default(); client.connect().await?; client .login_user(DEFAULT_ROOT_USERNAME, DEFAULT_ROOT_PASSWORD) .await?; - init_system(&client).await; + init_system(&client).await?; produce_messages(&client).await?; Ok(()) } -async fn init_system(client: &IggyClient) { +async fn init_system(client: &IggyClient) -> Result<(), IggyError> { match client.create_stream(STREAM_NAME).await { Ok(_) => info!("Stream was created."), - Err(_) => warn!("Stream already exists and will not be created again."), + Err(IggyError::StreamNameAlreadyExists(_)) => { + warn!("Stream already exists and will not be created again."); + } + Err(error) => return Err(error), } match client @@ -345,8 +370,12 @@ async fn init_system(client: &IggyClient) { .await { Ok(_) => info!("Topic was created."), - Err(_) => warn!("Topic already exists and will not be created again."), + Err(IggyError::TopicNameAlreadyExists(..)) => { + warn!("Topic already exists and will not be created again."); + } + Err(error) => return Err(error), } + Ok(()) } async fn produce_messages(client: &IggyClient) -> Result<(), Box> { @@ -408,15 +437,15 @@ At the first glance, it might look a bit more complicated than `send_messages` f - `topic_id` - the ID of the topic (numeric or string) from which we want to poll the messages. -- `partition_id` - the ID of the partition from which we want to poll the messages. The partition has to be specified for the regular `Consumer`, while for the `ConsumerGroup` it's **ignored** (`None` value), as the server will automatically assign the partition to the consumer from the group. +- `partition_id` - the ID of the partition from which we want to poll the messages. Specify the partition for a regular `Consumer`. For a joined `ConsumerGroup`, pass `None` to let the SDK select a partition from the assignments it synchronizes with the server. An explicit partition is also accepted, but the server checks that the member owns it. -- `consumer` - the type of the consumer (kind + ID), either the default `Consumer` means the standalone client which does the message polling on its own, independently of the other consumers (unless they would use the same ID), or the `ConsumerGroup` which might be used to create the group of consumers sharing the common identifier - this is especially useful in the case of the horizontal scaling, where we want to ensure, that the same (and only one) consumer, will poll the messages from the specific partition, and there will be no overlap with the other consumers from the same group. For example, when scaling out (by adding more instances) the group of payment processing microservices, we probably don't want the multiple instances to process the same payment. +- `consumer` - the type of the consumer (kind + ID), either the default `Consumer` means the standalone client which does the message polling on its own, independently of the other consumers (unless they would use the same ID), or the `ConsumerGroup` which might be used to create the group of consumers sharing the common identifier - this is especially useful in the case of the horizontal scaling, where we want to ensure, that the same (and only one) consumer, will poll the messages from the specific partition, and there will be no overlap with the other consumers from the same group. For example, when scaling out (by adding more instances) the group of payment processing microservices, we probably don't want the multiple instances to process the same payment. Group assignment coordinates polling, but application side effects still need idempotency across retries and rebalances. -- `strategy` - the way in which we want to poll the messages. The default one being `Offset` (underlying `PollingKind` enum) means that we will start polling the messages from the particular offset provided in the `value` field - it's on the client, to keep track of the most recent offset. On the other hand, we could also use the different kind, for example `Next`, which means, that the next messages will be returned to the client, depending on the so-called `consumer offset` value stored on the server side. The client may not need to track the offset on its own anymore, but instead, call `store_offset()` to save it on the server (e.g. after each processed message or the whole batch), or make use of `auto_commit: true`, to automatically commit the offset on the server side, once the messages are fetched (this one results in the so called *at-most-once* delivery mode). +- `strategy` - the way in which we want to poll the messages. The default one being `Offset` (underlying `PollingKind` enum) means that we will start polling the messages from the particular offset provided in the `value` field - it's on the client, to keep track of the most recent offset. On the other hand, we could also use the different kind, for example `Next`, which means, that the next messages will be returned to the client, depending on the so-called `consumer offset` value stored on the server side. The client may not need to track the offset on its own anymore, but instead, call `store_consumer_offset()` to save it on the server (e.g. after each processed message or the whole batch), or use `auto_commit: true` for server-managed offset advancement. Auto-commit is best-effort: a poll does not wait for the offset to commit, and a follower-served poll does not replicate it. Messages can be replayed after failure. The offset can also advance before the application processes the response, so use an explicit offset store after processing when that distinction matters. - `count` - amount of the messages that the consumer would like to receive in the single response from the server. -- `auto_commit` - whether the consumer offset should be automatically committed on the server side, once the messages are fetched. +- `auto_commit` - whether to request server-managed offset advancement when messages are fetched, with the best-effort behavior described above. Next, let's take a look at the following method responsible for polling the messages based on the specified interval. @@ -445,9 +474,9 @@ async fn consume_messages(client: &IggyClient) -> Result<(), Box> { continue; } - offset += polled_messages.messages.len() as u64; for message in polled_messages.messages { handle_message(&message)?; + offset = message.header.offset + 1; } sleep(interval).await; } @@ -469,7 +498,12 @@ const PARTITION_ID: u32 = 0; #[tokio::main] async fn main() -> Result<(), Box> { - tracing_subscriber::fmt::init(); + tracing_subscriber::fmt() + .with_env_filter( + tracing_subscriber::EnvFilter::try_from_default_env() + .unwrap_or_else(|_| "info".into()), + ) + .init(); let client = IggyClient::default(); client.connect().await?; client @@ -510,9 +544,9 @@ async fn consume_messages(client: &IggyClient) -> Result<(), Box> { continue; } - offset += polled_messages.messages.len() as u64; for message in polled_messages.messages { handle_message(&message)?; + offset = message.header.offset + 1; } sleep(interval).await; } diff --git a/src/components/architecture-diagrams.tsx b/src/components/architecture-diagrams.tsx index b6c8cbfe73..e97563d220 100644 --- a/src/components/architecture-diagrams.tsx +++ b/src/components/architecture-diagrams.tsx @@ -37,11 +37,11 @@ export function MessageFlowDiagram() { const steps = [ { label: "Client", desc: "Client sends messages via TCP/QUIC/WS/HTTP", icon: "M12 2C6.48 2 2 6.48 2 12s4.48 10 10 10 10-4.48 10-10S17.52 2 12 2zm-1 17.93c-3.95-.49-7-3.85-7-7.93 0-.62.08-1.21.21-1.79L9 15v1c0 1.1.9 2 2 2v1.93z" }, - { label: "Listener", desc: "Transport listener receives the request on a shard thread", icon: "M21 3L3 10.53v.98l6.84 2.65L12.48 21h.98L21 3z" }, + { label: "Listener", desc: "Shard 0 accepts connections; the connection owner decodes requests", icon: "M21 3L3 10.53v.98l6.84 2.65L12.48 21h.98L21 3z" }, + { label: "Stream", desc: "Resolve the stream ID or name from local metadata", icon: "M4 6h16v2H4zm0 5h16v2H4zm0 5h16v2H4z" }, + { label: "Topic", desc: "Resolve the topic and target partition; compression is not applied", icon: "M3 18h6v-2H3v2zM3 6v2h18V6H3zm0 7h12v-2H3v2z" }, { label: "Router", desc: "Request routed to owning shard via IggyNamespace hash", icon: "M12 2C6.48 2 2 6.48 2 12s4.48 10 10 10 10-4.48 10-10S17.52 2 12 2zm1 15h-2v-2h2v2zm0-4h-2V7h2v6z" }, - { label: "Stream", desc: "Stream lookup by ID (metadata read from left-right)", icon: "M4 6h16v2H4zm0 5h16v2H4zm0 5h16v2H4z" }, - { label: "Topic", desc: "Topic lookup within stream, compression applied", icon: "M3 18h6v-2H3v2zM3 6v2h18V6H3zm0 7h12v-2H3v2z" }, - { label: "Partition", desc: "Messages buffered in the partition journal (PartitionJournal)", icon: "M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zm0 16H5V5h14v14z" }, + { label: "Partition", desc: "The partition primary admits and replicates the write through VSR", icon: "M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zm0 16H5V5h14v14z" }, { label: "Segment", desc: "Flushed to .log file via vectored I/O (io_uring)", icon: "M14 2H6c-1.1 0-2 .9-2 2v16c0 1.1.9 2 2 2h12c1.1 0 2-.9 2-2V8l-6-6zm4 18H6V4h7v5h5v11z" }, ]; @@ -126,7 +126,7 @@ export function ShardDiagram() { id: 0, label: "Shard 0 (Coordinator)", color: "var(--color-fd-primary)", - features: ["Binds all listeners: TCP, QUIC, HTTP, WS", "Replica plane listener", "Metadata plane (left-right write handle)", "QUIC + TCP-TLS terminate here", "Hands plaintext TCP/WS to peers (fd transfer)"], + features: ["Binds all listeners: TCP, QUIC, HTTP, WS", "Replica plane listener", "Metadata plane (left-right write handle)", "QUIC + TCP-TLS + WSS + HTTP terminate here", "Hands plaintext TCP/WS to peers (fd transfer)"], partitions: ["P0", "P3", "P6"], }, { @@ -174,7 +174,7 @@ export function ShardDiagram() { compio runtime
- io_uring (4096 ops) + io_uring (default capacity: 4096)
@@ -213,8 +213,8 @@ export function ShardDiagram() { Inter-shard communication

- Shards communicate via crossfire bounded mpsc channels. Metadata mutations route to Shard 0, the only shard that commits; peers hold left-right read handles. - Partition ops route to the owning shard via the lock-free papaya::HashMap<IggyNamespace, PartitionLocation>. + Shards communicate via crossfire bounded mpsc channels. Metadata mutations route to Shard 0, the only shard that commits metadata; peers hold left-right read handles. + CPU labels and partition assignments above are illustrative. Partition ops route to the owning shard via the lock-free papaya::HashMap<IggyNamespace, PartitionLocation>.

@@ -239,7 +239,7 @@ export function IoUringComparison() { ))}
- Files are "always ready" for epoll. Tokio uses a blocking thread pool (up to 512 threads) for file I/O. + Regular files cannot be registered with epoll. Tokio uses a blocking pool for file I/O (512 threads by default, configurable).
@@ -259,7 +259,7 @@ export function IoUringComparison() { ))}
- Both SQ and CQ are lock-free ring buffers shared between user space and kernel. No syscall per I/O in the hot path. + SQ and CQ are shared ring buffers. Batching amortizes submission and completion syscalls across operations.
@@ -373,8 +373,8 @@ export function MessageHeaderDiagram() { const fields = [ { name: "checksum", bytes: "0-8", size: 8, type: "u64", desc: "xxHash3 integrity checksum", color: "#ef4444" }, - { name: "id", bytes: "8-24", size: 16, type: "u128", desc: "Unique message ID (UUIDv4)", color: "#f59e0b" }, - { name: "offset", bytes: "24-32", size: 8, type: "u64", desc: "Sequential offset in partition", color: "#10b981" }, + { name: "id", bytes: "8-24", size: 16, type: "u128", desc: "Client-supplied 128-bit ID; Rust SDK generates UUIDv4 when omitted", color: "#f59e0b" }, + { name: "offset", bytes: "24-32", size: 8, type: "u64", desc: "Increasing offset within the partition", color: "#10b981" }, { name: "timestamp", bytes: "32-40", size: 8, type: "u64", desc: "Server-assigned timestamp", color: "#3b82f6" }, { name: "origin_ts", bytes: "40-48", size: 8, type: "u64", desc: "Client-provided timestamp", color: "#6366f1" }, { name: "hdrs_len", bytes: "48-52", size: 4, type: "u32", desc: "User headers length", color: "#8b5cf6" }, @@ -386,9 +386,9 @@ export function MessageHeaderDiagram() { return (
-

Message Header (64 bytes, little-endian)

+

Rust SDK Header (64 bytes, little-endian)

- Every message starts with this fixed-size header for efficient aligned reads. + This is the SDK byte representation. Wire and disk batches use a 48-byte per-message frame header.

@@ -454,7 +454,7 @@ export function BenchmarkChart() {

Latency Improvements: Tokio vs Thread-per-Core

- Relative latency comparison (lower is better). Thread-per-core with io_uring vs Tokio work-stealing. + Selected historical results (lower is better): v0.5.0 vs v0.7.0 at approximately 1,000 MB/s per node. See the linked migration benchmark for workloads.

@@ -483,7 +483,7 @@ export function BenchmarkChart() { export function NamespacePacking() { const bitGroups = [ - { label: "unused (12 bits)", bits: 12, color: undefined, range: "63..52" }, + { label: "zero for partitions (12 bits)", bits: 12, color: undefined, range: "63..52" }, { label: "stream", bits: 20, color: "#f59e0b88", range: "51..32", sub: "20 bits" }, { label: "topic", bits: 12, color: "#3b82f688", range: "31..20", sub: "12 bits" }, { label: "partition", bits: 20, color: "#10b98188", range: "19..0", sub: "20 bits" }, @@ -499,7 +499,7 @@ export function NamespacePacking() {

IggyNamespace Bit Packing (u64)

- Stream, topic, and partition IDs are packed into a single u64 for efficient hashing and shard routing. + Stream, topic, and partition IDs are packed into a single u64 for efficient hashing and shard routing. Bit 63 is reserved for the separate metadata consensus group.

@@ -761,7 +761,7 @@ export function ConsumerGroupViz() {

Consumer Group

- Each partition is assigned to exactly one consumer. When a consumer joins or leaves, partitions are rebalanced. + Within this group, each partition has at most one member permitted to poll it. Joins and leaves trigger rebalancing; pending handoffs can temporarily pause polling.

From dfed33d87b08bf16a831d3e8acbf89e9c5604a3a Mon Sep 17 00:00:00 2001 From: hubcio Date: Fri, 11 Sep 2026 01:52:31 +0200 Subject: [PATCH 03/13] fix(docs): align server docs with 0.9.0 Server guidance drifted from configuration, authentication, storage and deployment behavior. Correct the verified claims and examples, qualify historical benchmarks, and document safe result metadata. Validate source startup, all benchmark kinds, Docker and Helm examples, and built pages in both themes. --- content/docs/server/benchmarking.mdx | 14 +++-- content/docs/server/configuration.mdx | 66 +++++++++++++----------- content/docs/server/docker.mdx | 23 +++++---- content/docs/server/introduction.mdx | 21 +++++--- content/docs/server/networking.mdx | 22 ++++---- content/docs/server/security.mdx | 26 +++++----- content/docs/server/storage-engine.mdx | 16 +++--- content/docs/server/topic-options.mdx | 28 ++++++---- src/components/architecture-diagrams.tsx | 12 ++--- 9 files changed, 127 insertions(+), 101 deletions(-) diff --git a/content/docs/server/benchmarking.mdx b/content/docs/server/benchmarking.mdx index 65b4fe0fa7..59f9770c9c 100644 --- a/content/docs/server/benchmarking.mdx +++ b/content/docs/server/benchmarking.mdx @@ -5,14 +5,18 @@ description: "How Iggy is benchmarked, the tooling that ships with it, and the p **Benchmarks should be the first-class citizens**. We believe that performance is crucial for any system, and we strive to provide the best possible performance for our users. Please check, why we believe that the **[transparent benchmarking](https://iggy.apache.org/blogs/2025/02/17/transparent-benchmarks)** is so important. -We've also built the **[benchmarking platform](https://benchmarks.iggy.apache.org)** where anyone can upload the benchmarks and compare the results with others. This is the another open-source project available [here](https://github.com/apache/iggy/tree/master/core/bench/dashboard). +We've also built the **[benchmarking platform](https://benchmarks.iggy.apache.org)** where you can browse published benchmarks and compare results. This is the another open-source project available [here](https://github.com/apache/iggy/tree/master/core/bench/dashboard). ![Benchmarking Platform](/img/bench_platform.png) +*Historical dashboard screenshot showing a 0.5.0 run, not 0.9.0 measurements.* + Iggy comes with a built-in benchmarking tool, `iggy-bench`. It's written in Rust and uses the `tokio` runtime for asynchronous I/O, mimicking the example client applications, so you can use it to estimate the performance of the server in your environment. It is part of the [core repository](https://github.com/apache/iggy/tree/master/core/bench) and lives in the `core/bench` directory. ![Bench CLI](/img/bench_cli.png) +*Historical CLI screenshot. Server-start and cleanup flags shown there have been removed; use the commands below and the current `iggy-bench --help`.* + ## Running benchmarks First build the project in release mode: @@ -79,7 +83,7 @@ Producer and consumer counts default to six. Pinned workloads also default to si cargo r --bin iggy-bench -r -- balanced-producer-and-consumer-group tcp ``` -7. End-to-end producing consumer (`e2e`): each task produces and then consumes its own messages, measuring the full round trip: +7. End-to-end producing consumer (`e2e`): each task alternates producing and consuming, measuring latency from the messages' producer timestamps: ```bash cargo r --bin iggy-bench -r -- end-to-end-producing-consumer tcp @@ -109,11 +113,13 @@ Each transport subcommand accepts a trailing `output` subcommand that persists t cargo r --bin iggy-bench -r -- pinned-producer tcp output -o performance_results --identifier my-host ``` -`-o/--output-dir` defaults to `performance_results`. Optional flags (`--remark`, `--gitref`, extra info) annotate the run. You can inspect persisted results with the in-repo report and runner crates (`core/bench/report`, `core/bench/runner`), and browse or compare them in the dashboard (`core/bench/dashboard`), which also powers the public [benchmarking platform](https://benchmarks.iggy.apache.org). A prebuilt dashboard image is available: `docker pull apache/iggy-bench-dashboard`. +When the server address uses `localhost` or `127.0.0.1`, the saved command includes non-secret environment settings from the server configuration catalog. Credentials and unknown `IGGY_` variables are omitted. + +`-o/--output-dir` defaults to `performance_results`. Optional flags (`--remark`, `--gitref`, extra info) annotate the run. Generated charts use the in-repo report library (`core/bench/report`). The runner (`core/bench/runner`) executes benchmarks across Git revisions. You can browse or compare results in the dashboard (`core/bench/dashboard`), which also powers the public [benchmarking platform](https://benchmarks.iggy.apache.org). A prebuilt dashboard image is available: `docker pull apache/iggy-bench-dashboard:edge`. ## Performance -The server is thread-per-core and shared-nothing, built on `io_uring` (via `compio`), with shard and CPU pinning configurable under `[sharding]`. Throughput and latency depend heavily on hardware, transport, and payload shape (`messages-per-batch * message-size`). Run the benchmarks on your own hardware, or browse current, dated results on the [benchmarking platform](https://benchmarks.iggy.apache.org). +The server is thread-per-core and shared-nothing, built on `io_uring` on Linux (via `compio`), with shard and CPU pinning configurable under `[sharding]`. Throughput and latency depend heavily on hardware, transport, and payload shape (`messages-per-batch * message-size`). Run the benchmarks on your own hardware, or browse current, dated results on the [benchmarking platform](https://benchmarks.iggy.apache.org). ## Prepare the host and topic policies diff --git a/content/docs/server/configuration.mdx b/content/docs/server/configuration.mdx index a5982da182..a550393b1e 100644 --- a/content/docs/server/configuration.mdx +++ b/content/docs/server/configuration.mdx @@ -32,7 +32,7 @@ Configuration is resolved in three layers. Later layers win: 2. **Config file**: the path in `IGGY_CONFIG_PATH`, or `core/server/config.toml` resolved against the current working directory. A missing file is only a warning. The server continues on embedded defaults. 3. **Environment variables**: any `IGGY_`-prefixed override. -Before the environment is read, the server loads a `.env` file from the working directory, or from the path named by `IGGY_ENV_PATH`. +Before the environment is read, the server loads a `.env` file from the working directory or its parents, or from the path named by `IGGY_ENV_PATH`. Existing environment values take precedence over the `.env` file. After boot the server writes the effective configuration, including the addresses it actually bound, to `{path}/runtime/current_config.toml` with the default runtime subdirectory. @@ -41,15 +41,15 @@ After boot the server writes the effective configuration, including the addresse Every configuration key can be overridden with an `IGGY_` variable. The name is the TOML path, uppercased, with dots turned into underscores: ```bash -IGGY_TCP_ADDRESS=0.0.0.0:8090 # [tcp] address -IGGY_NODE_ADVERTISED_ADDRESS=iggy-1 # [node] advertised_address -IGGY_HTTP_ENABLED=true # [http] enabled -IGGY_PATH=/var/lib/iggy # root path -IGGY_LOGGING_LEVEL=debug # [logging] level -IGGY_SHARDING_CPU_ALLOCATION=4 # [sharding] cpu_allocation +export IGGY_TCP_ADDRESS=0.0.0.0:8090 # [tcp] address +export IGGY_NODE_ADVERTISED_ADDRESS=iggy-1 # [node] advertised_address +export IGGY_HTTP_ENABLED=true # [http] enabled +export IGGY_PATH=/var/lib/iggy # root path +export IGGY_LOGGING_LEVEL=debug # [logging] level +export IGGY_SHARDING_CPU_ALLOCATION=4 # [sharding] cpu_allocation ``` -Two variables live outside the config schema: `IGGY_ROOT_USERNAME` and `IGGY_ROOT_PASSWORD` set the root credentials, always as a pair. **Only the first creation** of the root user reads them. On an existing data directory the stored root user is recovered unchanged. +Two variables live outside the config schema: `IGGY_ROOT_USERNAME` and `IGGY_ROOT_PASSWORD` set the root credentials, always as a pair. They initialize the root user **only at first creation**. On an existing data directory the stored root user is recovered unchanged, but supplied environment credentials must still pass validation. ### Secrets @@ -57,6 +57,7 @@ Five values are secret-flagged: `http.jwt.encoding_secret`, `http.jwt.decoding_s ### Validation at boot +- Array-valued environment settings replace the corresponding TOML array. For an indexed array such as `cluster.nodes`, supply every entry and its required fields through environment variables; a single field override does not merge into the existing TOML roster. - Unknown top-level TOML fields and unknown server `IGGY_` environment names **reject startup**. Some nested tables can still ignore unknown fields, so compare the effective configuration with the intended settings. - The old `[system]` table and its environment mappings are rejected. Removed placeholders such as `archive_expired` and `recreate_missing_state` must be removed, not set to `false`. - Several sections validate relationships between keys (QUIC windows, sharding shutdown budgets, metadata journal sizing, partition transfer floors). A violation aborts boot with an error naming the keys. The constraints are listed with their sections below. @@ -64,12 +65,12 @@ Five values are secret-flagged: `http.jwt.encoding_secret`, `http.jwt.decoding_s ## Command-line flags -`iggy-server` accepts exactly three flags: +`iggy-server` accepts these three startup options, plus `--help` (`-h`) and `--version` (`-V`): | Flag | Description | |------|-------------| | `--fresh`, `-f` | Delete the configured data directory (`local_data` by default, see `IGGY_PATH`) before boot and start on empty state. In cluster mode this wipes **this replica only**; it rejoins and refills by state transfer from the others. Wiping a quorum at the same time can destroy committed data. Do not put `--fresh` in a service unit: it would re-transfer the whole dataset on every restart. | -| `--with-default-root-credentials` | Set `IGGY_ROOT_USERNAME` and `IGGY_ROOT_PASSWORD` to `iggy` unless they are already present in the environment. Only the first creation of the root user reads these values. Development only. | +| `--with-default-root-credentials` | Set `IGGY_ROOT_USERNAME` and `IGGY_ROOT_PASSWORD` to `iggy` unless they are already present in the environment. These values initialize the root user only at first creation. Development only. | | `--replica-id ` | Identify this node within `cluster.nodes`. Required when `cluster.enabled = true`; the value must match exactly one `replica_id` in the roster. | ## Relocated configuration keys @@ -109,8 +110,8 @@ The tables below list every section with its shipped defaults. In cluster mode, |-----|---------|-------------| | `enabled` | `true` | Serve the HTTP REST API. | | `address` | `"127.0.0.1:3000"` | Bind address and port. | -| `max_request_size` | `"2 MB"` | Maximum request body size. | -| `web_ui` | `false` | Serve the embedded Web UI at `/ui`. Requires a server built with the `iggy-web` feature; without it, `true` logs a warning and the server continues. | +| `max_request_size` | `"2 MB"` | Maximum request body size, at most `"256 MiB"`. Keep it at or below `message_bus.max_message_size` for multi-replica topics so an admitted batch fits replica frames. | +| `web_ui` | `false` | Serve the embedded Web UI at `/ui`. Requires the `iggy-web` feature and static assets built with `npm --prefix web ci` and `npm --prefix web run build:static` before compiling the server. Without the feature, `true` logs a warning; missing assets return 404. | HTTP sessions hold live server-side session state (they count against `metadata.clients_table_max`). Consumer group management (create, get, delete) is available over HTTP. Group membership (join, leave) is not and needs a stateful transport. @@ -140,17 +141,17 @@ In cluster mode, followers forward control-plane requests (streams, topics, user | `access_token_expiry` | `"1 h"` | Access token lifetime. | | `clock_skew` | `"5 s"` | Tolerance for clock differences during validation. | | `not_before` | `"0 s"` | Time before which a token is not valid. | -| `encoding_secret` | `""` (empty) | Signing key. Empty means a secure random secret is generated on each server start. | -| `decoding_secret` | `""` (empty) | Verification key. Same empty-means-random behavior. | +| `encoding_secret` | `""` (empty) | Signing key. If only one secret is configured, it is used for both signing and verification. See the fallback rules below. | +| `decoding_secret` | `""` (empty) | Verification key. If both secrets are set, they are used as supplied and must agree for self-issued tokens to verify. | | `use_base64_secret` | `false` | Treat the configured secrets as base64-encoded. | -There is **no default secret**. With the secrets left empty, each server start mints a random signing key, so issued tokens **do not survive a restart** and are valid only on the node that issued them. If you configure a secret, set it through the environment (`IGGY_HTTP_JWT_ENCODING_SECRET` / `IGGY_HTTP_JWT_DECODING_SECRET`), never commit it, and use a long random value. +There is **no default secret**. When both secrets are empty and cluster authentication supplies no key, each server start mints a random signing key, so issued tokens **do not survive a restart** and are valid only on the node that issued them. If you configure a secret, set it through the environment (`IGGY_HTTP_JWT_ENCODING_SECRET` / `IGGY_HTTP_JWT_DECODING_SECRET`), never commit it, and use a long random value. -In cluster mode the secret has an extra role: a configured secret, identical on every node, makes bearer tokens valid cluster-wide and activates follower-to-primary HTTP forwarding. With `cluster.auth` enabled the JWT key is instead derived from the cluster PSK. Without either, tokens are node-local and forwarding stays disabled. +In cluster mode the secret has an extra role: a configured secret, identical on every node, makes bearer tokens valid cluster-wide and activates follower-to-primary HTTP forwarding. When both JWT secrets are empty and `cluster.auth` is enabled, the JWT key is derived from the cluster PSK. Explicit JWT secrets take precedence. Without either, tokens are node-local and forwarding stays disabled. #### `[[http.jwt.trusted_issuers]]` -Opt-in trust of external token issuers for application-to-application authentication. With none configured, the listener accepts only self-issued HS256 tokens. +Opt-in trust of external token issuers for application-to-application authentication. With none configured, the listener accepts only self-issued tokens using the configured HMAC algorithm (`HS256`, `HS384`, or `HS512`). ```toml [[http.jwt.trusted_issuers]] @@ -171,7 +172,7 @@ Enabling an issuer opens an outbound JWKS fetch that is reachable before a token | `enabled` | `true` | Expose Prometheus metrics. | | `endpoint` | `"/metrics"` | Metrics path. Must start with `/`. | -The metrics route **requires authentication** like every other read: a missing or invalid bearer credential is rejected with 401. Any authenticated user may scrape. There's no extra RBAC rule. Scrapers present a JWT or a personal access token as the bearer: +The metrics route **requires authentication**: a missing or invalid bearer credential is rejected with 401. Any authenticated user may scrape. There's no extra RBAC rule. Scrapers present a JWT or a personal access token as the bearer: ```yaml scrape_configs: @@ -288,8 +289,8 @@ Set `path` before any TOML table header. `[runtime] path = "runtime"` and `[logg | `path` | `"logs"` | Log directory, relative to the root `path`. | | `level` | `"info"` | Filter directive in `RUST_LOG` syntax: simple levels or directives like `"warn,server=debug,iggy=trace"`. The `RUST_LOG` environment variable always takes precedence. | | `file_enabled` | `true` | Write logs to file as well as stdout. | -| `max_file_size` | `"500 MB"` | Size at which a log file rotates. `0` means one unbounded file, which disables size-based rotation. | -| `max_total_size` | `"4 GB"` | Total log budget; oldest files are deleted first. `0` means unlimited archives. Time-based rotation still applies. | +| `max_file_size` | `"500 MB"` | Size at which a log file rotates. `0` disables both size-based and hourly rotation. | +| `max_total_size` | `"4 GB"` | Total log budget; oldest files are deleted first. `0` disables the total-size cleanup limit; the rolling appender still caps archives at 100000. Retention and, when `max_file_size` is nonzero, hourly rotation still apply. | | `rotation_check_interval` | `"1 h"` | How often rotation status is checked. Avoid values below 1 s. | | `retention` | `"7 days"` | How long log files are kept. Avoid values below 1 s. | @@ -304,8 +305,8 @@ Set `path` before any TOML table header. `[runtime] path = "runtime"` and `[logg | Key | Default | Description | |-----|---------|-------------| -| `enabled` | `true` | Use the pre-allocated buffer pool. | -| `size` | `"4 GiB"` | Total pool memory. Minimum 512 MiB; must be a multiple of 4096 (page size). | +| `enabled` | `true` | Reuse aligned buffers through the memory pool. Buffers are allocated on demand. | +| `size` | `"4 GiB"` | Pool allocation budget, not a process memory limit. When the pool cannot supply a buffer within this budget, allocation continues outside it. Minimum 512 MiB; must be a multiple of 4096 (page size). | | `bucket_capacity` | `8192` | Maximum buffers per bucket. Must be a power of two; minimum 128. | The pool has 28 buckets with buffer sizes from 4 KiB to 512 MiB. @@ -314,7 +315,7 @@ The pool has 28 buckets with buffer sizes from 4 KiB to 512 MiB. | Key | Default | Description | |-----|---------|-------------| -| `cleaner_enabled` | `true` | Run the segment cleaner. It deletes the oldest **sealed** segments of topics with a finite `message_expiry` or `max_topic_size`, per partition, best-effort. The active segment is never touched. | +| `cleaner_enabled` | `true` | Run the segment cleaner. It deletes the oldest **sealed** segments of topics with a finite `message_expiry` or `max_topic_size`, per partition, best-effort. The active segment is never touched, and stored consumer/group offsets can hold back deletion. | | `interval` | `"1 m"` | Cleaner run interval. | ### `[heartbeat]` @@ -324,9 +325,9 @@ The pool has 28 buckets with buffer sizes from 4 KiB to 512 MiB. | `enabled` | `true` | Verify client heartbeats. | | `interval` | `"30 s"` | Expected heartbeat interval. | -When enabled, a connection that sends nothing (no request, no PING) for 1.2 x `interval` (36 s at the default) **and** still holds a consumer group membership has its session released, so its groups rebalance off it. A connection holding no membership is left alone and reaped when its socket closes. +When enabled, a connection that sends nothing (no request, no PING) for 1.2 x `interval` (36 s at the default) **and** still holds a consumer group membership has its session released, so its groups rebalance off it. The verifier checks once per `interval`, so eviction can occur later than the staleness threshold. A connection holding no membership is left alone and reaped when its socket closes. -The Rust, Go, Python, Node, and async Java SDKs ping automatically every 5 s, well inside the staleness window, but only from a connected high-level client: the Rust and async Java pingers are armed by `connect()`, so a session that logs in without it never pings. The blocking Java and C# SDKs have **no automatic heartbeat**, only a manual ping. Wherever nothing pings, an idle consumer group member is evicted, and only the application can keep it alive (ping) or bring it back (reconnect). +The Rust, Go, Python, Node, async Java, and C# TCP clients ping automatically every 5 s by default, well inside the staleness window. Call `connect()` to start the heartbeat in the Rust, Go, Python, and async Java high-level clients; logging in without it does not start their pingers. The blocking Java client has **no automatic heartbeat**, only a manual ping. Wherever nothing pings, an idle consumer group member is evicted, and only the application can keep it alive (ping) or bring it back (reconnect). ### `[telemetry]` @@ -345,7 +346,8 @@ The Rust, Go, Python, Node, and async Java SDKs ping automatically every 5 s, we |-----|---------|-------------| | `cpu_allocation` | `"numa:auto"` | Number of shards and their CPU affinity. See syntaxes below. | | `pin_cores` | `true` | Pin shard threads to dedicated cores, drawn from the process's allowed CPU set (cooperates with systemd `AllowedCPUs=` and container cpusets). Set `false` on hosts where the server shares cores with other workloads. | -| `inbox_capacity` | `1024` | Per-shard inter-shard inbox capacity. Bounded by design; size for the consensus working set plus peak client-reply fan-out. Raising `[metadata]` or `[partition]` `prepare_queue_depth` raises the capacity needed here. | +| `inbox_capacity` | `1024` | Per-shard inbox capacity for consensus, connection setup, and reconciliation. Raising `[metadata]` or `[partition]` `prepare_queue_depth` raises the capacity needed here. | +| `reply_inbox_capacity` | `1024` | Separate per-shard channel for forwarded client replies. Size for peak reply fan-out; dropped replies have no bus retransmission. | | `shutdown_drain_timeout` | `"10 s"` | Per-shard bus drain budget on shutdown. Slow-fsync hosts may need more. | | `shutdown_poll_interval` | `"50 ms"` | Poll cadence for the shutdown flag. Must be less than or equal to `shutdown_drain_timeout`. | | `shutdown_join_timeout` | `"30 s"` | Hard deadline for joining shard threads at exit; a wedged shard is abandoned with an error log. Must be at least `shutdown_drain_timeout`. | @@ -377,9 +379,12 @@ Partition storage and consensus tunables share this table. Unlike `[metadata]` ( | Key | Default | Description | |-----|---------|-------------| -| `prepare_queue_depth` | `32` | Uncommitted produce and consumer-offset ops in flight per partition. Submits past it spill into a request queue of twice this depth; once both are full the server drops the request without a reply and the client retries on its own timeout. Must be between 1 and 127. | +| `prepare_queue_depth` | `32` | Uncommitted produce and consumer-offset ops in flight per partition. Submits past it spill into a request queue of twice this depth; once both are full the server rejects the request with `TransientNotAccepted`. A lost rejection reply can still make the client wait for its timeout. Must be between 1 and 127. | | `validate_checksum` | `true` | Re-hash batches read from segment storage and report a mismatch instead of serving them. | | `wal_bytes_max` | `"256 MiB"` | Active WAL plus queued/in-flight prepare budget per multi-replica partition when either topic policy is `persisted`. A 4 KiB multiple, from 128 MiB + 8 KiB through 4 GiB. Checkpointing reclaims history only after materialized state is synchronized. Temporary rewrites require extra disk space. Environment override: `IGGY_PARTITION_WAL_BYTES_MAX`. | +| `dedup_clients_max` | `4096` | Client request watermarks retained per partition to deduplicate retries. At capacity, the client with the oldest latest commit is evicted and loses that coverage. Between 1 and 65536. | +| `consumer_offsets_max` | `4096` | Durable offset keys admitted per partition, counted separately for standalone consumers and groups. Existing keys remain writable at the limit; new keys are rejected with `TooManyConsumerOffsets`. Between 1 and 262144. | +| `offset_reservation_lease` | `65536` | Offsets reserved ahead in the superblock for single-replica partitions. Crash recovery skips the unused reservation to avoid reusing acknowledged offsets. Multi-replica groups ignore it. Between 1 and 16777216. | | `evicted_ring_capacity` | `4096` | Entries retained per multi-replica partition for journal repair after a peer rejoins. Must be between 1 and 65536. Single-replica partitions retain nothing. | | `evicted_ring_bytes_max` | `"16 MiB"` | Byte ceiling for the evicted ring; whichever ring cap trips first evicts. At most `"256 MiB"`. | | `transfer_served_cache_bytes_max` | `"2176 MiB"` | Byte budget, **per shard**, for segment payloads kept resident to serve state-transfer chunk requests. The default fits two sealed segments at the 1 GiB ceiling, each with one max-message overshoot. Serving concurrency is `floor(this / max(transfer_artifact_bytes_max, 1 GiB + 64 MiB))`, minimum one; boot warns when it drops below two. At most `"64 GiB"`. | @@ -392,7 +397,7 @@ Tunables for the internal bus that ships consensus traffic between replicas and | Key | Default | Description | |-----|---------|-------------| | `max_batch` | `256` | Messages coalesced into one `writev(2)` call. Hard upper bound 512 (`IOV_MAX/2` on Linux). | -| `max_message_size` | `"64 MiB"` | Wire-level cap on a single framed message. Coupled to `partition.transfer_artifact_bytes_max` and `websocket.max_message_size` (see those keys). | +| `max_message_size` | `"64 MiB"` | Wire-level cap on a single framed message, at most `"256 MiB"`. Values above `"64 MiB"` exceed the Go SDK's frame limit. Coupled to `partition.transfer_artifact_bytes_max` and `websocket.max_message_size` (see those keys). | | `peer_queue_capacity` | `256` | Bound on the per-peer queue. `cluster.repair_chunk_max` must stay strictly below it. | | `reconnect_period` | `"5 s"` | Interval between outbound reconnect attempts to peers. | | `close_peer_timeout` | `"2 s"` | Per-peer close drain budget before force-cancellation. | @@ -437,12 +442,15 @@ Cluster mode is configured here but documented in [Clustering](/docs/clustering/ | `request_start_view_retransmit_interval` | `"1s"` | Re-request cadence for the current view's StartView. | | `view_probe_attempts_max` | `5` | Unanswered probes a recovering replica tolerates before electing on its recovered log. Between 1 and 100. | | `repair_retry_interval` | `"1s"` | Re-request cadence for a stalled journal-repair stream. | +| `repair_gap_debounce_interval` | `"1s"` | How long a committed journal gap waits before repair starts, floored at 500 ms. Nonzero. Separate from retries of an already-open repair stream. | | `repair_chunk_max` | `128` | Prepares served per repair round. Must stay strictly below `message_bus.peer_queue_capacity`. Between 1 and 1024. | +| `superblock_wedged_fatal_timeout` | `"2m"` | Exit if a metadata or partition superblock remains unwritable past this window. `"0"` leaves it fenced indefinitely; nonzero values must be at least `"30s"`. | Durations are **rounded down** to the consensus 10 ms tick. Values under 10 ms become one tick. Sub-sections: - `[cluster.auth]`: replica-to-replica authentication (PSK plus BLAKE3 keyed-MAC handshake). `shared_secret` must be at least 32 bytes of CSPRNG output, identical on every node. Prefer `IGGY_CLUSTER_AUTH_SHARED_SECRET`. `previous_shared_secret` enables rolling key rotation. +- `[cluster.coordinator]`: `skip_shard_zero_for_replicas = true` and `skip_shard_zero_for_clients = false` control connection placement when more than one shard runs. - `[cluster.tls]`: TLS 1.3 on the replica port. Requires `cluster.auth.enabled`. The PSK authenticates the peer, TLS supplies confidentiality. -- `[[cluster.nodes]]`: the full roster, byte-identical on every node. Each entry has `name`, `ip`, `replica_id`, and `ports` (`tcp`, `quic`, `http`, `websocket`, `tcp_replica`). In cluster mode `ports` is the single source of listener ports: every enabled transport needs an explicit per-node port or the server refuses to start. `advertised_address` and per-CIDR `advertised_addresses` selectors control what clients are told to dial. +- `[[cluster.nodes]]`: the full roster, byte-identical on every node. Each entry has `name`, `ip`, `replica_id`, and `ports` (`tcp`, `quic`, `http`, `websocket`, `tcp_replica`). In cluster mode `ports` is the single source of listener ports: `tcp` and `tcp_replica` ports are always required, and every other enabled transport also needs an explicit per-node port or the server refuses to start. `advertised_address` and per-CIDR `advertised_addresses` selectors control what clients are told to dial. diff --git a/content/docs/server/docker.mdx b/content/docs/server/docker.mdx index c46b89a36a..f39ce49577 100644 --- a/content/docs/server/docker.mdx +++ b/content/docs/server/docker.mdx @@ -5,7 +5,7 @@ description: "Run the Iggy server from the official Docker images, and deploy it ## Docker -You can easily run the Iggy server with Docker - the official images can be found [here](https://hub.docker.com/r/apache/iggy), simply type `docker pull apache/iggy`. +You can easily run the Iggy server with Docker - the official images can be found [here](https://hub.docker.com/r/apache/iggy), use `docker pull apache/iggy:edge` while preparing for 0.9.0, or `apache/iggy:0.9.0` once released. These properties of the published image matter for deployment: @@ -20,7 +20,7 @@ The examples use a permissive syscall profile and unlimited locked memory for de ```yaml services: iggy: - image: apache/iggy:latest + image: apache/iggy:edge container_name: iggy restart: unless-stopped cap_add: @@ -59,7 +59,7 @@ docker run -d --name iggy \ -e IGGY_NODE_ADVERTISED_ADDRESS=localhost \ -p 8090:8090 -p 3000:3000 \ -v iggy:/app/local_data \ - apache/iggy:latest + apache/iggy:edge ``` ### Why these capabilities? @@ -105,17 +105,17 @@ Helm charts for Kubernetes deployment are available in the [repository](https:// ### Quick start ```bash -helm install iggy ./helm/charts/iggy +helm install iggy ./helm/charts/iggy --set server.image.tag=edge ``` ### Chart components -- **Server Deployment** - runs `apache/iggy` with the pod security context the server needs: seccomp profile `Unconfined` (for `io_uring`) plus the `IPC_LOCK` capability (for memory locking). Listener addresses are set to `0.0.0.0` via `server.env`, and the data volume mounts at `/app/local_data`. The chart supplies `IGGY_NODE_ADVERTISED_ADDRESS` as the in-cluster Service DNS name; override it with `server.advertisedAddress` when clients arrive through a LoadBalancer or an Ingress. Images that predate the setting, including the `0.7.0` pinned by default, log the variable as unknown and start anyway. -- **Server Service** - exposes the `http` (3000), `quic` (8080), and `tcp` (8090) ports. WebSocket is not exposed by the chart. +- **Server Deployment** - runs `apache/iggy` with the pod security context the server needs: seccomp profile `Unconfined` (for `io_uring`) plus the `IPC_LOCK` capability (for memory locking). Listener addresses are set to `0.0.0.0` via `server.env`, and the data volume mounts at `/app/local_data`. The chart supplies `IGGY_NODE_ADVERTISED_ADDRESS` as the in-cluster Service DNS name; override it with `server.advertisedAddress` when clients arrive through a LoadBalancer or an Ingress. The server image tag defaults to the chart's `appVersion` (`0.9.0-edge.6` in this source revision). The quick start explicitly selects `edge`; use `server.image.tag=0.9.0` after that release is available. +- **Server Service** - exposes the `http` (3000), `quic` (8080/UDP), `tcp` (8090), and `websocket` (8092) ports. - **Web UI Deployment + Service** - a separate `apache/iggy-web-ui` deployment on port 3050, enabled by default (`ui.enabled`). -- **Secret** - root user credentials from `server.users.root` (default `iggy`/`changeit`). Point `existingSecret` at your own Secret in production. +- **Secret** - root user credentials from `server.users.root` (default `iggy`/`changeit`). Point `server.users.root.existingSecret.name` at your own Secret in production; `usernameKey` and `passwordKey` select its keys. - **PersistentVolumeClaim** - storage for `/app/local_data`, **disabled by default** (`server.persistence.enabled`), 8Gi when enabled. -- **ServiceAccount**, **HPA**, **Ingress** - the usual optional plumbing, for both server and UI. +- **ServiceAccount** and **Ingress** - a shared ServiceAccount and separate optional server/UI ingresses. The chart has no HPA template. Server `replicaCount` must not exceed 1; cluster mode uses one release per node, each with its own replica ID and storage. - **ServiceMonitor** - Prometheus scrape config (optional). The `/metrics` endpoint requires a bearer credential like every other read, so wire a token (e.g. a personal access token stored in a Secret) into `server.serviceMonitor.authorization` - an unauthenticated scrape gets 401. ### Key values @@ -131,11 +131,12 @@ server: advertisedAddress: "" image: repository: apache/iggy - tag: "0.7.0" + tag: "" # Falls back to Chart.yaml appVersion ports: http: 3000 quic: 8080 tcp: 8090 + websocket: 8092 users: root: username: iggy @@ -169,8 +170,8 @@ securityContext: resources: {} ``` -Customize the values file for your environment and deploy with: +Save the excerpt as `my-values.yaml`, customize it for your environment, and deploy with: ```bash -helm install iggy ./helm/charts/iggy -f my-values.yaml +helm install iggy ./helm/charts/iggy -f my-values.yaml --set server.image.tag=edge ``` diff --git a/content/docs/server/introduction.mdx b/content/docs/server/introduction.mdx index bdac48bdb9..adbfbfed7d 100644 --- a/content/docs/server/introduction.mdx +++ b/content/docs/server/introduction.mdx @@ -3,17 +3,17 @@ title: Introduction description: "What the Iggy server does, and where its releases and Docker images are published." --- -Iggy server is the most important part of the system as it's responsible for handling all the incoming connections, managing the data and providing the API for the clients. The server is written in Rust and can be run on any platform that supports it. +Iggy server is the most important part of the system as it's responsible for handling all the incoming connections, managing the data and providing the API for the clients. The server is written in Rust. It uses `io_uring` on Linux and a polling backend on macOS. -The releases are published to GitHub and can be found [here](https://github.com/apache/iggy/tags). The official Docker images can be found [here](https://hub.docker.com/r/apache/iggy), simply type `docker pull apache/iggy`. +The releases are published to GitHub and can be found [here](https://github.com/apache/iggy/tags). The official Docker images can be found [here](https://hub.docker.com/r/apache/iggy), use `docker pull apache/iggy:edge` while preparing for 0.9.0, or `apache/iggy:0.9.0` once released. -If you compile the source code in release mode, the longer compilation time comes from [LTO](https://doc.rust-lang.org/rustc/linker-plugin-lto.html) enabled in the `[profile.release]` section of the workspace [Cargo.toml](https://github.com/apache/iggy/blob/master/Cargo.toml). +If you compile the source code in release mode, linking takes longer because [LTO](https://doc.rust-lang.org/cargo/reference/profiles.html#lto) is enabled in the `[profile.release]` section of the workspace [Cargo.toml](https://github.com/apache/iggy/blob/master/Cargo.toml). ## Running the server -One `iggy-server` binary serves both the single-node and the clustered deployment. The loaded configuration decides which one you get. The server accepts three CLI flags: +One `iggy-server` binary serves both the single-node and the clustered deployment. The loaded configuration decides which one you get. The server accepts these startup flags, plus `--help` and `--version`: | Flag | Purpose | |------|---------| @@ -21,18 +21,23 @@ One `iggy-server` binary serves both the single-node and the clustered deploymen | `--with-default-root-credentials` | Set the root credentials to `iggy`/`iggy` on first start, unless `IGGY_ROOT_USERNAME`/`IGGY_ROOT_PASSWORD` are already set. Development only. | | `--replica-id ` | Select this node's entry in `cluster.nodes`. Required when `cluster.enabled = true`. See [Clustering](/docs/clustering/vsr). | -Configuration comes from the TOML file named by `IGGY_CONFIG_PATH`. Without one, the server boots on the defaults embedded in the binary. Any key can be overridden with an `IGGY_`-prefixed environment variable, and a `.env` file in the working directory (or the file named by `IGGY_ENV_PATH`) is loaded at startup. See [Configuration](/docs/server/configuration) for the full reference. +Configuration comes from the TOML file named by `IGGY_CONFIG_PATH`, or `core/server/config.toml` relative to the working directory. If that file is missing, the server uses the defaults embedded in the binary. Any key can be overridden with an `IGGY_`-prefixed environment variable, and a `.env` file in the working directory or its parents (or the file named by `IGGY_ENV_PATH`) is loaded at startup. See [Configuration](/docs/server/configuration) for the full reference. -When no root credentials are provided on the very first start, the server generates a random root password and prints it to the log. That's the *only* time it can be read. +When no root credentials are provided on the first single-node start, the server generates a random root password and prints it to the log. That's the *only* time it can be read. The HTTP API endpoints can be found in [server.http](https://github.com/apache/iggy/blob/master/core/server/server.http) file, which can be used with [REST Client](https://marketplace.visualstudio.com/items?itemName=humao.rest-client) extension for VS Code. In order to see the detailed logs from the server, run it with `RUST_LOG=trace` environment variable. -To seed the example data, start the server with known credentials, then run the seeder from the root of the repository: +To seed the example data, start a development server with known credentials from the root of the repository: ```bash cargo run --bin iggy-server -- --fresh --with-default-root-credentials +``` + +In another terminal at the repository root: + +```bash cargo run --bin data-seeder-tool ``` @@ -40,4 +45,4 @@ The seeder logs in as `iggy`/`iggy` by default. Pass `--username` and `--passwor ## Authentication -Only the ping liveness probe and the login handshake itself (username/password, personal access token, or HTTP token refresh, all of which prove a credential) are served without an authenticated session. Every other request requires one and is subject to [permissions](/docs/server/security): fetching server stats, for example, needs the `read_servers` permission (the root user has it), and even the Prometheus `/metrics` scrape **must present a bearer credential**. A session is created by logging in with the user's credentials or personal access token and is valid for the duration of the connection or until the user logs out. Over the HTTP API, authentication is done by providing the `Authorization` header with the `Bearer` token. +For broker API commands, only the ping liveness probe and the login handshake itself (username/password, personal access token, or HTTP token refresh, all of which prove a credential) are served without an authenticated session. Every other request requires one and is subject to [permissions](/docs/server/security): fetching server stats, for example, needs the `read_servers` permission (the root user has it), and even the Prometheus `/metrics` scrape **must present a bearer credential**. A stateful connection authenticates by logging in with the user's credentials or personal access token. Logout, disconnection, or heartbeat eviction releases its session. Over the HTTP API, authentication is done by providing the `Authorization` header with a `Bearer` JWT or personal access token. HTTP CORS preflight responses and embedded `/ui` static assets are public; the UI's broker API calls still require authentication. diff --git a/content/docs/server/networking.mdx b/content/docs/server/networking.mdx index 9deb6b9199..7fcd92c4cf 100644 --- a/content/docs/server/networking.mdx +++ b/content/docs/server/networking.mdx @@ -22,7 +22,7 @@ Every request and reply on the stateful transports starts with a fixed 256-byte ## Connection handling across shards -The server runs one shard per core (thread-per-core, shared-nothing). All listeners (every client transport) bind on shard 0 only. Shard 0's coordinator hands accepted plaintext TCP and WebSocket connections to peer shards round-robin by transferring the file descriptor at accept time. From then on the connection lives entirely on its shard. QUIC, TCP with TLS, and HTTP terminate on shard 0, because their per-connection state cannot move between threads. Sockets **never migrate** after accept: an operation that touches a partition owned by another shard rides the internal message bus instead. +The server runs one thread and runtime per selected shard (thread-per-core, shared-nothing), with CPU selection and optional pinning controlled by `[sharding]`. All listeners (every client transport) bind on shard 0 only. Shard 0's coordinator hands accepted plaintext TCP and WebSocket connections to peer shards round-robin by transferring the file descriptor at accept time. From then on the connection lives entirely on its shard. QUIC, TCP with TLS, WebSocket with TLS, and HTTP terminate on shard 0, because their per-connection state cannot move between threads. Sockets **never migrate** after accept: an operation that touches a partition owned by another shard rides the internal message bus instead. ## TCP @@ -64,7 +64,7 @@ key_file = "" A few of these are deliberate rather than arbitrary: -- `max_concurrent_bidi_streams = 1`: the SDK opens a fresh stream per command and the server accepts one at a time, so a connection carries one in-flight command. Raising the cap does **not** make handlers concurrent. +- `max_concurrent_bidi_streams = 1`: the Rust SDK opens a fresh stream per command and the server accepts one at a time, so a connection carries one in-flight command. Raising the cap does **not** make handlers concurrent. - `initial_mtu = "1200 B"` is the QUIC minimum. MTU discovery probes upward automatically, and boot rejects values below 1200. - `stream_receive_window` is the per-stream slice of `receive_window`, equal to it because there is a single stream. Lower it if you ever raise the stream cap. - `keep_alive_interval` is one third of `max_idle_timeout`, so two consecutive lost keep-alives fit before the idle timer closes the connection. @@ -73,20 +73,20 @@ A few of these are deliberate rather than arbitrary: The HTTP API is built on [axum](https://github.com/tokio-rs/axum) and provides a standard REST interface. It includes JWT authentication, CORS configuration, and optional TLS. -HTTP is the most accessible protocol but has the highest overhead due to JSON serialization and the stateless nature of HTTP (no persistent connections for consumer groups). You can find all the available endpoints in the [server.http](https://github.com/apache/iggy/blob/master/core/server/server.http) file. +HTTP is the most accessible protocol but has the highest overhead due to JSON serialization and the stateless nature of HTTP (no persistent connections for consumer groups). You can find request examples in the [server.http](https://github.com/apache/iggy/blob/master/core/server/server.http) file. The HTTP server also hosts: -- **Prometheus metrics** at `/metrics` (endpoint configurable via `[http.metrics]`). The route authenticates like every other read: scrapers must present a bearer credential (JWT or personal access token). `/ping` is the *only* route that requires no credential at all. -- **Embedded Web UI** at `/ui`, which requires `web_ui = true` in `[http]` (**off by default**) on a server built with the `iggy-web` feature (part of the default build). +- **Prometheus metrics** at `/metrics` (endpoint configurable via `[http.metrics]`). The route authenticates like every other read: scrapers must present a bearer credential (JWT or personal access token). The `/ping` API route, CORS preflight responses, and enabled `/ui` static assets are public. Login and refresh routes prove credentials in the request body. +- **Embedded Web UI** at `/ui`, which requires `web_ui = true` in `[http]` (**off by default**) on a server built with the `iggy-web` feature (part of the default build) and the Web UI static assets. Build the assets before compiling the server; see [Configuration](/docs/server/configuration). ## WebSocket -WebSocket provides bidirectional streaming over HTTP upgrade. Iggy uses its own `compio-ws` implementation that bridges tungstenite's poll-based model with compio's completion-based I/O, reading frames through a buffered stream with a 128 KiB base buffer that can grow to a 64 MiB cap. +WebSocket provides bidirectional streaming over HTTP upgrade. Iggy uses the `compio-ws` crate to connect tungstenite framing to compio's completion-based I/O. The default WebSocket read buffer is 128 KiB and the default maximum message size is 64 MiB; these are separate settings. The `[websocket]` section exposes the frame-tuning knobs (`read_buffer_size`, `write_buffer_size`, `max_write_buffer_size`, `max_message_size`, `max_frame_size`, `accept_unmasked_frames`). See [Configuration](/docs/server/configuration). TLS for WebSocket has its own `[websocket.tls]` section. -**Benchmark comparison** (AWS i3en.3xlarge, fsync per message, 4 producers, 40M messages): +**Historical benchmark comparison** from the [WebSocket implementation article](/blogs/2025/11/17/websocket-io-uring) (AWS i3en.3xlarge, 4 producers or consumers, 40M messages, 1,000 messages per batch, fsync enabled). These measurements predate 0.9.0: - Producer avg latency: TCP 2.61ms vs WebSocket 3.43ms (+31%) - Consumer avg latency: TCP 0.70ms vs WebSocket 1.44ms (+106%) @@ -100,9 +100,9 @@ Each transport configures TLS in its own section, and the `self_signed` semantic | TCP | `[tcp.tls]` | `self_signed = true` generates an ephemeral certificate only while `cert_file` does not exist; an existing PEM pair is loaded instead. | | WebSocket | `[websocket.tls]` | Same load-or-generate rule as TCP. | | QUIC | `[quic.certificate]` | TLS is mandatory (part of the QUIC spec). `self_signed = true` **always** generates an ephemeral certificate and ignores `cert_file`/`key_file`, logging a warning if the files exist. | -| HTTP | `[http.tls]` | No `self_signed` option: HTTPS requires real `cert_file`/`key_file`. | +| HTTP | `[http.tls]` | No `self_signed` option: HTTPS requires certificate and key files in `cert_file`/`key_file`. | -With `self_signed = false`, `cert_file` and `key_file` must both exist. For production, provide proper certificate files everywhere. Ephemeral certificates **change on every start** and cannot be verified by clients. +With `self_signed = false`, `cert_file` and `key_file` must both exist. For production, provide proper certificate files everywhere. Ephemeral certificates **change on every start**, so a trust configuration pinned to one generated certificate does not survive a restart. ## Heartbeat @@ -112,9 +112,9 @@ enabled = true interval = "30 s" ``` -A connection that sends nothing (no request, no ping) for 1.2x the interval has its session released so its consumer groups rebalance off it. Only connections holding a consumer-group membership are evicted. Others are left alone until their socket closes. +The verifier checks once per interval and releases a session whose last heartbeat is more than 1.2x the interval old, so its consumer groups rebalance off it. Requests and pings refresh the heartbeat. Only connections holding a consumer-group membership are evicted. Others are left alone until their socket closes. -Most SDKs (Rust, Go, Python, Node, async Java) ping automatically every 5 seconds from a connected high-level client, well inside the resulting 36-second window. The blocking Java and C# SDKs have **no automatic heartbeat**, so an idle consumer-group member there must ping manually or reconnect after eviction. +The Rust, Go, Python, Node, async Java and C# TCP clients have automatic heartbeats, normally every 5 seconds once connected. This is below the default 36-second inactivity threshold; the periodic verifier means eviction does not occur at an exact 36-second deadline. The blocking Java client has no automatic heartbeat, so an idle consumer-group member must ping manually or reconnect after eviction. ## Cluster networking diff --git a/content/docs/server/security.mdx b/content/docs/server/security.mdx index fa6386005c..c88137f468 100644 --- a/content/docs/server/security.mdx +++ b/content/docs/server/security.mdx @@ -11,34 +11,34 @@ Iggy supports two authentication mechanisms: ### Username and password -Users authenticate with a username and password via `login_user()`. Passwords are hashed using **Argon2id** (a memory-hard hashing algorithm). On first startup, the server generates a random password for the `root` user and logs it to the console. You can override this by setting environment variables: +Users authenticate with a username and password, for example via the Rust SDK's `login_user()`. Passwords are hashed using **Argon2id** (a memory-hard hashing algorithm). On first single-node startup, the server generates a random password for the initial root account (username `iggy` by default) and writes it to the logs. A first cluster boot requires explicit root credentials. You can override this by setting environment variables: ```bash -IGGY_ROOT_USERNAME=iggy -IGGY_ROOT_PASSWORD=my-secret-password +export IGGY_ROOT_USERNAME=iggy +export IGGY_ROOT_PASSWORD=my-secret-password ``` Or use the `--with-default-root-credentials` flag for development (sets root credentials to `iggy`/`iggy`). -**Important**: once the data directory exists, environment variable credentials are **ignored**. To reset credentials, you must use the `--fresh` flag (which deletes all data). +These variables initialize the root account only when it is first created. Supplied values are still validated on later starts, but they do not replace the stored credentials. Use the password-change API or `iggy user password` with the current password to change an existing account. The `--fresh` flag deletes all data; it is not needed for a password change. The root user **cannot be deleted**, and its permissions are fixed: it always holds every permission. ### Personal Access Tokens (PAT) -PATs provide programmatic access with optional expiry. Each user can have up to `max_tokens_per_user` (default 100) active tokens. Tokens are **hashed before storage** and can be revoked at any time. +PATs provide programmatic access with optional expiry. Each user can have up to `max_tokens_per_user` (default 100) stored tokens, including expired tokens until the cleaner removes them. Tokens are **hashed before storage** and can be revoked at any time. ```bash # Create a PAT via CLI -iggy -u iggy -p secret pat create my-token 7d +IGGY_TOKEN=$(iggy --quiet -u iggy -p my-secret-password pat create my-token 7d) # Use the PAT for authentication -iggy -t my-token-value stream list +iggy -t "$IGGY_TOKEN" stream list ``` An automatic cleaner removes expired tokens at a configurable interval. -Only the ping liveness probe and the login endpoints themselves are served without authentication. **Everything else** - including the Prometheus `/metrics` scrape - requires an authenticated session or bearer credential. +Broker API commands require an authenticated session or bearer credential, except for ping and the login/refresh flows that establish or prove a credential. The Prometheus `/metrics` scrape requires a JWT or personal access token. HTTP CORS preflight responses and embedded `/ui` static assets are public; the UI's broker API calls still require authentication. ## Authorization @@ -70,7 +70,7 @@ The global permissions and the operations they unlock: Two kinds of implication apply on top of the table: -- **Supersets**: every `manage_*` permission includes its `read_*` counterpart. In addition `manage_streams` includes `manage_topics`, `read_streams` includes `read_topics`, and `read_topics` includes `poll_messages`. +- **Supersets**: every `manage_*` permission includes its `read_*` counterpart. In addition `manage_streams` includes `manage_topics`, `read_streams` includes `read_topics`, and `read_topics` includes `poll_messages`. `manage_topics` also includes `send_messages`, so `manage_streams` permits sending too. - **Self-service**: an authenticated user can always read their own account, change their own password, and manage their own personal access tokens, without any of the user permissions above. ### Scoped permissions @@ -90,7 +90,7 @@ Permissions are checked from top to bottom: global, then stream, then topic. A p If a stream has no entry in the user's stream permissions, only global permissions apply to it. The same holds for topics within a stream. -For example, a user that may only consume from stream 42 needs no global permissions at all: grant stream-scoped `read_stream` and `poll_messages` on stream 42, and the user can read that stream, list its topics, and poll messages from any topic in it. Nothing else. +For example, a user that may only consume from stream 42 needs no global permissions at all: grant stream-scoped `read_stream` and `poll_messages` on stream 42, and the user can read that stream, list its topics, and poll messages from any topic in it. The read grant also permits consumer-group operations in that stream. For polling without those read and group permissions, grant only `poll_messages`. ## Transport encryption (TLS) @@ -101,9 +101,9 @@ Each transport configures TLS in its own section, and the `self_signed` semantic | TCP | `[tcp.tls]` | `self_signed = true` generates an ephemeral certificate only while `cert_file` does not exist; an existing PEM pair is loaded instead. | | WebSocket | `[websocket.tls]` | Same load-or-generate rule as TCP. | | QUIC | `[quic.certificate]` | TLS is mandatory (part of the QUIC spec). `self_signed = true` **always** generates an ephemeral certificate and ignores `cert_file`/`key_file`, logging a warning if the files exist. | -| HTTP | `[http.tls]` | No `self_signed` option: HTTPS requires real `cert_file`/`key_file`. | +| HTTP | `[http.tls]` | No `self_signed` option: HTTPS requires certificate and key files in `cert_file`/`key_file`. | -For production deployments, provide proper certificates via `cert_file` and `key_file`. Ephemeral certificates change on every start and cannot be verified by clients. See [Networking](/docs/server/networking) for the surrounding transport configuration. +For production deployments, provide proper certificates via `cert_file` and `key_file`. Ephemeral certificates change on every start, so a trust configuration pinned to one generated certificate does not survive a restart. See [Networking](/docs/server/networking) for the surrounding transport configuration. ## Data encryption at rest @@ -130,7 +130,7 @@ clock_skew = "5 s" Further keys (`valid_issuers`, `valid_audiences`, `not_before`, `use_base64_secret`, the signing secrets) are covered in [Configuration](/docs/server/configuration). -**Signing secrets**: `encoding_secret` and `decoding_secret` default to empty, which makes the server generate a secure random secret on every start. That's a safe single-node default with two consequences: issued tokens **die on restart**, and in a cluster each node signs with its own key, so bearers are node-local and follower-to-primary request forwarding stays disabled. For clusters, configure an identical secret on every node (prefer the `IGGY_HTTP_JWT_ENCODING_SECRET`/`IGGY_HTTP_JWT_DECODING_SECRET` environment variables over on-disk config), or enable `cluster.auth` so the JWT key derives from the cluster's shared PSK. +**Signing secrets**: `encoding_secret` and `decoding_secret` default to empty. Without `cluster.auth`, the server generates a secure random secret on every start. That's a safe single-node default with two consequences: issued tokens **die on restart**, and in a cluster each node signs with its own key, so bearers are node-local and follower-to-primary request forwarding stays disabled. For clusters, configure an identical secret on every node (prefer the `IGGY_HTTP_JWT_ENCODING_SECRET`/`IGGY_HTTP_JWT_DECODING_SECRET` environment variables over on-disk config), or enable `cluster.auth` so the JWT key derives from the cluster's shared PSK. **Refresh tokens**: `POST /users/refresh-token` re-issues an access token from a still-valid one presented in the request body, answering the same identity shape as login, so HTTP clients can extend a session without re-sending credentials. diff --git a/content/docs/server/storage-engine.mdx b/content/docs/server/storage-engine.mdx index a9fcca05ff..21aa67ee37 100644 --- a/content/docs/server/storage-engine.mdx +++ b/content/docs/server/storage-engine.mdx @@ -3,7 +3,7 @@ title: Storage Engine description: "The segmented append-only log, and how streams, topics, partitions and segments map onto files on disk." --- -Iggy's storage engine is built around the concept of a **segmented append-only log**. Every piece of data flows through a well-defined hierarchy: System -> Streams -> Topics -> Partitions -> Segments. This page covers how data is stored, indexed, flushed, recovered, and cleaned up on disk. +Iggy's storage engine is built around the concept of a **segmented append-only log**. Message data follows a hierarchy: System -> Streams -> Topics -> Partitions -> Segments. This page covers how data is stored, indexed, flushed, recovered, and cleaned up on disk. @@ -42,7 +42,7 @@ local_data/ └── 00000000000016000000.index ``` -Stream, topic, and partition ids are numeric and **0-based**. Each partition directory holds pairs of `.log` and `.index` files. The filename is the start offset of the segment's first message, zero-padded to 20 digits. Next to the segments, every partition keeps its own superblock pair (replica identity and consensus state for that partition) and an `offsets/` tree for consumer offset storage. Multi-replica partitions also keep a prepare WAL when either topic durability policy is `persisted`. +Stream, topic, and partition ids are numeric and **0-based**. Each partition directory holds pairs of `.log` and `.index` files. The filename is the segment's start offset, zero-padded to 20 digits. Recovery can create an empty active segment at a reserved offset beyond the last stored message. Next to the segments, every partition keeps its own superblock pair (replica identity and consensus state for that partition) and an `offsets/` tree for consumer offset storage. Multi-replica partitions also keep a prepare WAL when either topic durability policy is `persisted`. ## Segmented log @@ -129,13 +129,13 @@ Disk reads are verified before they reach a consumer: validate_checksum = true ``` -With `validate_checksum = true` (the default), every batch a disk poll reads is re-hashed and compared against its stored checksum. A mismatch **fails the poll closed**, so a segment damaged at rest is reported instead of served. Setting it to `false` skips the re-hash and serves whatever decodes, which can hand a consumer bytes provably not the ones written. Only disable it with a corruption guard elsewhere in the stack. +With `validate_checksum = true` (the default), every batch a disk poll reads is re-hashed and compared against its stored checksum. A mismatch stops the disk walk and logs an error on the server. The consumer receives any valid prefix already read, or an ordinary empty poll if none was read; the protocol does not report the checksum failure to the consumer. Repeated polls can therefore wait indefinitely at damaged data. Setting it to `false` skips the re-hash and serves whatever decodes, which can hand a consumer bytes provably not the ones written. Only disable it with a corruption guard elsewhere in the stack. -Polls serve stored batch records as-is (a reply may be a server-sliced view of a larger stored batch), so there is no re-encoding on the read path. +Binary polls reuse the stored batch layout. A reply may slice a larger stored batch and rewrite its header, and at-rest encryption requires decryption before the reply. HTTP additionally serializes the result as JSON. ## Memory pool -Iggy includes a custom memory pool to eliminate allocation overhead on the hot path. The pool has **28 buckets** with buffer sizes from 4 KiB up to 512 MiB (non-uniform spacing, denser around common message sizes, with sizes above 2 MiB rounded to hugepage-friendly steps). Components request a buffer from the appropriate bucket and return it when done. +Iggy includes a custom memory pool to reuse buffers on the hot path. The pool has **28 buckets** with buffer sizes from 4 KiB up to 512 MiB (non-uniform spacing, denser around common message sizes, with sizes above 2 MiB rounded to hugepage-friendly steps). Components request a buffer from the appropriate bucket and return it when done. ```toml [memory_pool] @@ -144,11 +144,11 @@ size = "4 GiB" # Total pool size (minimum 512 MiB, multiple of the 40 bucket_capacity = 8192 # Buffers per bucket (power of 2, minimum 128) ``` -This avoids heap allocations during message processing and enables zero-copy message passing between internal components. +Buffers are allocated lazily. A pool miss can allocate outside the pool, so message processing is not allocation-free. Internal components can pass ownership of existing buffers without copying their contents. ## Retention and cleanup -Two independent retention policies exist, both **per-topic creation options** (see [Topic options](/docs/server/topic-options)). The segment cleaner enforces them: +Two independent retention policies exist, both **per-topic options configurable at creation or update** (see [Topic options](/docs/server/topic-options)). The segment cleaner enforces them: ```toml [data_maintenance.messages] @@ -160,7 +160,7 @@ interval = "1 m" # default **Time-based retention** (`message_expiry`): sealed segments whose newest message is older than the expiry are deleted. -Both policies can be active at once. They only ever touch sealed segments. The active segment is **never deleted**, even if its messages have expired. +Both policies can be active at once. They only ever touch sealed segments. Deletion also stops at the minimum committed consumer or consumer-group offset, and pending persistence checkpoints can defer it. The active segment is **never deleted by retention**, even if its messages have expired. ## Metadata plane diff --git a/content/docs/server/topic-options.mdx b/content/docs/server/topic-options.mdx index e32c42b8d5..d2833fce4c 100644 --- a/content/docs/server/topic-options.mdx +++ b/content/docs/server/topic-options.mdx @@ -11,23 +11,23 @@ Options are key-value pairs sent with `CreateTopic`. Unknown keys are **rejected | Option | Default | Constraints | Description | |--------|---------|-------------|-------------| -| `max_topic_size` | unlimited | | Delete the oldest sealed segments once the topic grows past this size. | -| `message_expiry` | none | | Delete sealed segments older than this. | +| `max_topic_size` | unlimited | finite values at least `segment_size` | Per-partition limit on sealed segment bytes. Delete the oldest sealed segments once that partition exceeds it. | +| `message_expiry` | none | | Delete the oldest sealed segments whose newest message timestamp is older than this duration. | | `compression_algorithm` | `none` | `none` or `gzip` | Placeholder: stored and reported, no compression applied yet. | -| `segment_size` | 1 GiB | 512-byte multiple, at least 1 MiB, at most 1 GiB | Soft size limit per segment: a segment may close one whole batch past it. | +| `segment_size` | 1 GiB | 0 selects the default; otherwise a 512-byte multiple from 1 MiB through 1 GiB | Soft size limit per segment: a segment may close one whole batch past it. | | `durability` | `replicated` | `replicated` or `persisted` | Message completion policy. `persisted` requires recoverable stable-storage copies at the replication quorum before success. | | `consumer_offset_durability` | `replicated` | `replicated` or `persisted` | Completion policy for explicit consumer-offset stores and deletes, independent of message durability. | | `messages_required_to_save` | 1024 | non-zero, at most 16777216 | Flush the journal once it holds this many messages. | -| `size_of_messages_required_to_save` | 1 MiB | at most 1 GiB | Flush the journal once it holds this many bytes. Paired with the message count; whichever threshold trips first flushes. | +| `size_of_messages_required_to_save` | 1 MiB | 0 selects the default; at most 1 GiB | Flush the journal once it holds this many bytes. Paired with the message count; whichever threshold trips first flushes. | | `preallocate_segments` | `false` | `segment_size` x partitions at most 64 GiB per create | Reserve each segment's bytes up front where the filesystem supports it. | -Both retention policies can be active at once. The active segment is **never touched**. Deletion is done by the server's segment cleaner (`[data_maintenance.messages]`, enabled by default). +Both retention policies can be active at once. The active segment is **never touched**, and its bytes are excluded from the size limit. A segment whose end offset exceeds the lowest stored consumer or consumer-group offset is retained; with no stored offsets, this barrier is absent. Deletion is done by the server's segment cleaner (`[data_maintenance.messages]`, enabled by default). Both durability policies write data to disk and default independently to `replicated`. Neither inherits the other. Flush thresholds schedule ordinary segment writes; required persistence, capacity pressure, or lifecycle operations can flush earlier. They do not weaken the `persisted` completion guarantee. Value forms are forgiving: byte sizes accept a raw number of bytes or a string like `"128 MiB"`, expiry accepts microseconds or a humantime string like `"7 days"`, booleans accept `true`/`false`. Create admission re-parses and re-encodes what you send, so a string `segment_size=128MiB` is stored as the number it names. -`preallocate_segments` reserves exactly `segment_size` of real disk per partition the moment the topic is created (and again as segments rotate). With the default 1 GiB segment size that's 1 GiB per partition up front, which is why it's opt-in and why one create is **capped at 64 GiB** of total reservation. +`preallocate_segments` requests a `segment_size` reservation for each partition's segment file when it is opened, including during topic creation and rotation. On Linux this reserves disk space without changing the file's logical length. Unsupported or failed reservations log a warning and fall back to ordinary allocation. With the default 1 GiB segment size, the request is 1 GiB per partition up front, which is why it's opt-in and why one create is **capped at 64 GiB** of requested reservation. ## Setting options @@ -46,7 +46,7 @@ In the CLI, both durability policies have named flags. `--set` is repeatable and SDKs expose typed durability values in their topic creation options. In Rust, set `TopicCreateOptions::durability` and `TopicCreateOptions::consumer_offset_durability` to `Durability::Replicated` or `Durability::Persisted`. The HTTP API takes `"durability"` and `"consumer_offset_durability"` as string values in the create body's `options` map. -Keys absent from the wire request are resolved by the admitting server and stored as **derived** entries. Typed SDKs can send their default durability values explicitly. `GetTopic` returns explicit and derived blocks, so the effective values and the provenance of the request remain visible. +Keys absent from the wire request are resolved by the admitting server and stored as **derived** entries. Typed SDKs can send their default durability values explicitly. Binary `GetTopic` responses carry explicit and derived blocks; HTTP reports an `explicit` flag per option. Both report option values and the provenance of the request. See the update limitation below. ## Create-only vs updatable @@ -60,19 +60,25 @@ Updates are **patches**: a key you don't send keeps its current value. The storage options (`segment_size`, `durability`, `consumer_offset_durability`, `messages_required_to_save`, `size_of_messages_required_to_save`, `preallocate_segments`) are **create-only**. A topic gets them at creation and keeps them. `UpdateTopic` rejects changes to either durability policy, so select both before creating the topic. +An update that explicitly sends `server_default` (zero) for `message_expiry` or `max_topic_size` has inconsistent reporting: the server retains the previous fixed response fields but stores zero in the corresponding option entries. The CLI sends these sentinels when those update arguments are omitted. Supply explicit expiry and size values when updating a topic; do not rely on zero to reset or preserve its retention settings. + ## Discovering the catalog Ask the server which keys it accepts, with their types, defaults, and descriptions: ```bash -# CLI iggy options topic +``` -# HTTP -GET /options/topic +For HTTP, set `IGGY_TOKEN` to a valid JWT or personal access token: + +```bash +curl --fail-with-body \ + -H "Authorization: Bearer $IGGY_TOKEN" \ + http://localhost:3000/options/topic ``` -SDKs expose the same call as `describe_options`. The scope is `topic`, `stream`, or `user`. +The Rust SDK exposes the same call as `describe_options`. The scope is `topic`, `stream`, or `user`. Discovery matters most on the binary transports: TCP, QUIC, and WebSocket carry **only an error code** for a rejected key, not its name, so the catalog is how a client finds out what this server supports. Over HTTP the error message names the offending key directly. diff --git a/src/components/architecture-diagrams.tsx b/src/components/architecture-diagrams.tsx index e97563d220..0284943136 100644 --- a/src/components/architecture-diagrams.tsx +++ b/src/components/architecture-diagrams.tsx @@ -280,7 +280,7 @@ export function SegmentVisualization() {

Partition Storage Layout

- Each partition contains a segmented log. Segments are sealed at 1 GiB and new ones created automatically. Click a segment to inspect its files. + Example partition at the default 1 GiB segment size. Rotation can exceed that size by one batch; message counts depend on payload and batch sizes. Click a segment to inspect its files.

@@ -343,7 +343,7 @@ export function SegmentVisualization() { .log
Message data (headers + payloads) - batch records, 48-byte frame per message + batch records, 48-byte header per message
@@ -356,8 +356,8 @@ export function SegmentVisualization() {
Offsets: {seg.startOffset.toLocaleString()} .. {seg.endOffset.toLocaleString()} - {seg.status === "sealed" && Read-only, safe to archive} - {seg.status === "active" && Accepting writes via vectored I/O} + {seg.status === "sealed" && Read-only segment} + {seg.status === "active" && Active; flushed with vectored I/O}
)} @@ -850,7 +850,7 @@ export function ServerEcosystem() {
Iggy Server - Thread-per-core + io_uring + Thread-per-core + io_uring (Linux)
{["TCP :8090", "QUIC :8080", "HTTP :3000", "WS :8092"].map((p) => ( {p} @@ -1213,7 +1213,7 @@ export function DocsHero() { Iggy Server - Thread-per-core + io_uring + Thread-per-core + io_uring (Linux)
{["TCP", "QUIC", "WS", "HTTP"].map((p) => ( {p} From 180bb023647c7ca904027beebe2af663df6d7083 Mon Sep 17 00:00:00 2001 From: hubcio Date: Fri, 11 Sep 2026 02:48:34 +0200 Subject: [PATCH 04/13] fix(docs): align CLI guidance with server 0.9.0 Document the matching source build, credential precedence and command limits. Correct retention and durability options, ignored ID flags, unsupported operations and executable examples. Verify the examples against the worktree server and a private keyring. Website build, typecheck, links, headers and both-theme renders pass. --- content/docs/cli/commands.mdx | 48 ++++++++++++++++++++--------------- content/docs/cli/start.mdx | 39 ++++++++++++++++++---------- 2 files changed, 54 insertions(+), 33 deletions(-) diff --git a/content/docs/cli/commands.mdx b/content/docs/cli/commands.mdx index 9b91b710ef..eecd875e0b 100644 --- a/content/docs/cli/commands.mdx +++ b/content/docs/cli/commands.mdx @@ -36,17 +36,17 @@ Commands: Conventions used throughout: -- Wherever a command takes a stream, topic, user, or consumer group ID, you can pass either the numeric ID or the name (`iggy topic get dev events` and `iggy topic get 1 1` are equivalent). +- Commands that address an existing stream, topic, user, or consumer group accept its numeric ID or name (for example, `iggy topic get dev events`). Numeric IDs identify the same resources only when those are the IDs assigned by the server. The numeric stream/topic IDs inside permission specifications are an exception: names are not accepted there. - Subcommands have single-letter aliases too: `c` (create), `d` (delete), `g` (get), `l` (list), `u` (update), `p` (purge). So `iggy s l` is `iggy stream list`. - Every `list` command accepts `-l, --list-mode ` (default: `table`). -- `-q, --quiet` suppresses stdout output. `-d, --debug ` writes verbose logs to a file. +- `-q, --quiet` suppresses ordinary status output; `stats` still prints its result, and `pat create` still prints an unstored token. Interactive password changes also print their confirmation. `-d, --debug ` writes verbose logs to a file. ## stream -Manage streams, the top-level containers for topics. Subcommands: `create`, `delete`, `update`, `get`, `list`, `purge`. +Manage streams, the top-level containers for topics. Subcommands: `create`, `delete`, `update`, `get`, `list`, `purge`. The legacy creation flags `stream create -s`, `topic create -t`, and `consumer-group create -g` are accepted but ignored; IDs are always assigned by the server. ```bash -# Create a stream (the server assigns the ID; use -s to pick one) +# Create a stream (the server assigns the ID) iggy stream create dev # List, inspect, rename @@ -63,19 +63,23 @@ iggy stream delete development ## topic -Manage topics within a stream. Subcommands: `create`, `delete`, `update`, `get`, `list`, `purge`. +Manage topics within a stream. Subcommands: `create`, `delete`, `update`, `get`, `list`, `purge`. These examples require an existing `dev` stream. ```text iggy topic create [OPTIONS] [MESSAGE_EXPIRY]... iggy topic update [OPTIONS] [MESSAGE_EXPIRY]... ``` -Compression is a **required positional**: `none` or `gzip`. Expiry is a human-readable duration (`7d`, `1day 12h`, `unlimited`). Omitting it uses the server default. +Compression is a **required positional**: `none` or `gzip`; it is stored as topic metadata, but compression is not applied. Expiry is a human-readable duration (`7d`, `1day 12h`, `unlimited`). Omitting expiry at creation selects no expiration. + +For updates, supply expiry and `--max-topic-size` explicitly. Omitted values are sent as `server_default` (zero): the server retains the previous expiry/size fields but reports zero in their option entries. See [Topic options](/docs/server/topic-options) for this inconsistency. | Flag | Description | |------|-------------| -| `-t, --topic-id ` | Explicit topic ID (create only; server assigns one by default) | -| `-m, --max-topic-size ` | Max topic size, e.g. `15GB`, `unlimited` (default: `server_default`) | +| `-t, --topic-id ` | Accepted at creation but ignored; the server assigns the ID | +| `-m, --max-topic-size ` | Retention size, e.g. `15GB`, `unlimited` (default: `server_default`; unlimited at creation) | +| `--durability ` | Message completion policy: `replicated` (default) or `persisted` (create only) | +| `--consumer-offset-durability ` | Independent offset-store/delete policy: `replicated` (default) or `persisted` (create only) | | `--set ` | Additional server-side option, repeatable (create only) | ```bash @@ -89,7 +93,7 @@ iggy topic list dev iggy topic get dev events # Update: name and compression are required positionals, in that order -iggy topic update dev metrics metrics gzip 60d +iggy topic update dev metrics metrics gzip 60d --max-topic-size unlimited iggy topic purge dev metrics iggy topic delete dev metrics @@ -109,16 +113,16 @@ iggy partition delete dev events 2 ## segment -Delete the oldest segments of a partition. +Delete up to the requested number of oldest sealed segments of a partition. The active segment is retained. ```bash -# Delete 3 segments from partition 1 of dev/events +# Delete up to 3 sealed segments from partition 1 of dev/events iggy segment delete dev events 1 3 ``` ## message -Send, poll, and flush messages. Subcommands: `send`, `poll`, `flush`. +Send and poll messages. Subcommands: `send`, `poll`, `flush`; the server does not implement `flush`. ### message send @@ -129,7 +133,7 @@ iggy message send [OPTIONS] [MESSAGES]... | Flag | Description | |------|-------------| | `-p, --partition-id ` | Send to a specific partition | -| `-m, --message-key ` | Route by message key, hashed to a partition client-side (mutually exclusive with `--partition-id`) | +| `-m, --message-key ` | Route by a 1-255 byte message key. Binary transports hash it client-side; HTTP resolves it server-side. Mutually exclusive with `--partition-id` | | `-H, --headers ` | Message headers, comma separated. Kinds: `raw`, `string`, `bool`, `int8`-`int128`, `uint8`-`uint128`, `float32`, `float64` | | `--input-file ` | Send messages from a binary file written by `poll --output-file` | @@ -165,7 +169,7 @@ iggy message poll [OPTIONS] <--offset |--first|--last|--next> ` | Number of messages to poll (default: 1) | -| `-a, --auto-commit` | Commit the consumer offset on the server after polling | +| `-a, --auto-commit` | Submit the current batch's last offset asynchronously before delivering the poll reply; not an acknowledged offset store | | `-c, --consumer ` | Consumer name or ID to poll as (default: `0`) | | `-s, --show-headers` | Include message headers in the output | | `--output-file ` | Append polled messages to a binary file instead of printing | @@ -188,9 +192,10 @@ iggy message send --input-file backup.bin --partition-id 1 dev events-replay ### message flush -Force a flush of the unsaved buffer of a partition to disk. With `-f, --fsync` the data is also fsynced. +`message flush` and `-f, --fsync` remain accepted by the CLI, but the server returns `FeatureUnavailable` on binary transports and has no HTTP flush route. This command does not flush data. Select `--durability persisted` when creating the topic to require stable-storage message acknowledgments. ```bash +# Returns FeatureUnavailable on the default TCP transport iggy message flush dev events 1 --fsync ``` @@ -211,6 +216,7 @@ iggy user status analytics-reader inactive # Change password; omit the passwords to be prompted securely iggy user password analytics-reader +# Or supply both passwords directly iggy user password analytics-reader Str0ngPass1 N3wPass2 iggy user delete analytics-reader @@ -267,6 +273,7 @@ Inspect the clients currently connected to the server. Clients are **connections ```bash iggy client list +# Use an ID from the list while that connection is still open iggy client get 42 ``` @@ -283,7 +290,7 @@ iggy cluster metadata Manage consumer groups of a topic. Subcommands: `create`, `delete`, `get`, `list`. ```bash -# Server assigns the group ID; use -g to pick one +# Server assigns the group ID iggy consumer-group create dev events reporting iggy consumer-group list dev events @@ -302,7 +309,7 @@ iggy consumer-offset get 1 dev events 1 # Offset of a consumer group iggy consumer-offset get reporting dev events 1 --kind consumer-group -# Rewind a consumer to offset 100 +# Set the stored offset to 100 (the partition must contain that offset) iggy consumer-offset set 1 dev events 1 100 ``` @@ -317,7 +324,7 @@ iggy context use production ## options -Print the server's option catalog for one scope: `topic`, `stream` or `user`. These are the keys the corresponding `create` command accepts via `--set KEY=VALUE`, with their kinds, defaults, and bounds. +Print the server's option catalog for one scope: `topic`, `stream` or `user`, with key kinds, defaults, and bounds. Only the topic catalog has entries, and only `topic create` exposes `--set KEY=VALUE` in the CLI. ```bash iggy options topic @@ -334,7 +341,7 @@ iggy ping -c 5 ## me -Show info about the current connection: client ID, user ID, server address, protocol. +Show info about the current connection: client ID, user ID, client address as seen by the server, protocol. Supported on TCP, QUIC and WebSocket; HTTP returns `FeatureUnavailable`. ```bash iggy me @@ -357,9 +364,10 @@ Collect server troubleshooting data into an archive, useful for support bundles. |------|-------------| | `-c, --compression ` | `stored`, `deflated`, `bzip2`, `zstd`, `lzma`, `xz` | | `-s, --snapshot-types ` | Space-separated subset of `filesystem_overview`, `process_list`, `resource_usage`, `test`, `server_logs`, `server_config`, `all` | -| `-o, --out-dir ` | Output directory for the snapshot file | +| `-o, --out-dir ` | Existing output directory for the snapshot file | ```bash +mkdir -p snapshots iggy snapshot --compression zstd --snapshot-types server_logs server_config --out-dir ./snapshots ``` diff --git a/content/docs/cli/start.mdx b/content/docs/cli/start.mdx index 330a431f6c..eacef4a2a2 100644 --- a/content/docs/cli/start.mdx +++ b/content/docs/cli/start.mdx @@ -11,16 +11,22 @@ This page covers installing the CLI, connecting to a server, authenticating, and ### Cargo -Install from crates.io with the Cargo package manager: +These docs target server 0.9.0 and the CLI built from the same source checkout. The CLI has its own version number. From the root of that checkout, install it with: ```bash -cargo install iggy-cli +cargo install --path core/cli --locked ``` -This builds and installs the `iggy` binary. If you have [cargo-binstall](https://github.com/cargo-bins/cargo-binstall), you can skip compilation and fetch a prebuilt release binary: +A published edge package is also available from crates.io. Version `0.14.0-edge.7` predates the new durability flags and CLI fixes documented here; use the source build for those commands: ```bash -cargo binstall iggy-cli +cargo install iggy-cli --version 0.14.0-edge.7 +``` + +This builds and installs the `iggy` binary. If you have [cargo-binstall](https://github.com/cargo-bins/cargo-binstall), it can fetch a matching prebuilt binary when one is available, otherwise it falls back to compilation: + +```bash +cargo binstall iggy-cli --version 0.14.0-edge.7 ``` ### Docker @@ -28,7 +34,7 @@ cargo binstall iggy-cli The official `apache/iggy` image ships the CLI alongside the server, installed as `/usr/local/bin/iggy`. The image entrypoint is the server, so override it to run the CLI: ```bash -docker run --rm -it --network host --entrypoint iggy apache/iggy -u iggy -p iggy ping +docker run --rm -it --network host --entrypoint iggy apache/iggy:edge -u iggy -p iggy ping ``` `--network host` works on **Linux**. On macOS and Windows, point the CLI at the host instead: `--tcp-server-address host.docker.internal:8090`. @@ -70,7 +76,7 @@ Each transport also exposes reconnection and tuning flags (retry counts, interva ## Authentication -Commands that talk to the server require credentials. The CLI resolves them in this order, **first match wins**: +Broker commands require credentials, except for `ping`. For commands other than `login`, the CLI resolves credentials in this order, **first match wins**: 1. A cached login session token (created by `iggy login`). 2. `-n, --token-name `: a personal access token stored in the platform keyring under that name. @@ -78,6 +84,8 @@ Commands that talk to the server require credentials. The CLI resolves them in t 4. `-u, --username ` with `-p, --password `. When `-p` is omitted, the CLI prompts for the password interactively (or reads one line from stdin when piped). 5. The `IGGY_USERNAME` and `IGGY_PASSWORD` environment variables (both must be set). +`iggy login` tries the supplied credentials before a cached session, so it can replace an expired login. Other commands fail if their cached token is rejected; they do not retry with the next credential source in the same invocation. + `-u`, `-t` and `-n` are **mutually exclusive**. Avoid passing the password inline with `-p`: it lands in your **shell history**. Prefer `iggy login`, the interactive prompt, or the environment variables. ```bash @@ -92,7 +100,7 @@ iggy stream list ### Login sessions -`iggy login` authenticates once and stores a session token in the platform's secure credential store: Secret Service on Linux and the BSDs, Keychain on macOS, Credential Manager on Windows. Subsequent commands use the cached token automatically. +`iggy login` authenticates once and stores a session token in the platform's secure credential store: Secret Service on Linux and the BSDs, Keychain on macOS, Credential Manager on Windows. Subsequent commands use the cached token automatically. This requires the default `login-session` feature and an available credential-store backend; Linux and BSD need a running Secret Service provider on the D-Bus session. ```bash # Login for 1 hour (default is 15 minutes; "none" disables expiry) @@ -108,7 +116,7 @@ iggy session status iggy logout ``` -`iggy session status` only checks whether a token exists in the local keyring. An expired token **still reports as active**. Run `iggy me` to verify the session against the server. +`iggy session status` only checks whether a token exists in the local keyring. An expired token **still reports as active**. On TCP, QUIC and WebSocket, run `iggy me` to verify the session against the server. HTTP does not support `me`; use an authenticated command such as `iggy stream list` with the required permissions. ### Personal access tokens @@ -125,8 +133,8 @@ iggy -n my-token stream list `--store-token` is mutually exclusive with an expiry: stored tokens **never expire**. They're also **namespaced per server address**, so a token stored for one server isn't visible when connecting to another. Alternatively, create a token with an expiry and pass its value with `-t`: ```bash -iggy -u iggy -p iggy pat create ci-token 7d -iggy -t stream list +iggy_cli_token=$(iggy -u iggy -p iggy -q pat create ci-token 7d) +iggy -t "$iggy_cli_token" stream list ``` ## Connection contexts @@ -153,16 +161,21 @@ iggy context show production iggy context delete production ``` -There's no `--context` flag: the active context is persistent state, switched with `iggy context use `. Flags passed on the command line override the corresponding values from the active context. +There's no `--context` flag: the active context is persistent state, switched with `iggy context use `. Flags passed on the command line override the corresponding values from the active context. Credential fields merge independently: supplying `--username` does not clear a context token, which still has higher authentication priority. -Contexts are stored in `contexts.toml`, and the active context name in `.active_context`, both under the Iggy home directory: `~/.iggy` by default, overridable with the `IGGY_HOME` environment variable. The `default` context always exists and **cannot be deleted or redefined**. Deleting the currently active context switches back to `default`. Context names may contain letters, digits, hyphens and underscores. +Contexts are stored in `contexts.toml`, and the active context name in `.active_context`, both under the Iggy home directory: `~/.iggy` by default, overridable with the `IGGY_HOME` environment variable. The `default` context always exists and **cannot be deleted or recreated through context commands**. Deleting the currently active context switches back to `default`. Context names may contain letters, digits, hyphens and underscores. Passwords and raw tokens supplied to `context create` are stored as plaintext in `contexts.toml` (owner-only permissions on Unix); use a stored token name when the credential should stay in the keyring. ## Shell completions Generate completions for bash, zsh, fish, elvish or powershell with `--generate`: ```bash -iggy --generate bash > /etc/bash_completion.d/iggy +iggy --generate bash > iggy_completion.bash +source iggy_completion.bash + +mkdir -p ~/.zfunc ~/.config/fish/completions iggy --generate zsh > ~/.zfunc/_iggy iggy --generate fish > ~/.config/fish/completions/iggy.fish ``` + +For zsh, add `~/.zfunc` to `fpath` before running `compinit` in your shell configuration. From d660c5761977cedb8c08684ed473880c4fc177b9 Mon Sep 17 00:00:00 2001 From: hubcio Date: Fri, 11 Sep 2026 03:45:53 +0200 Subject: [PATCH 05/13] fix(docs): align binary protocol with server 0.9.0 --- content/docs/binary-protocol/cluster.mdx | 36 ++++++++-------- content/docs/binary-protocol/commands.mdx | 27 ++++++++---- .../binary-protocol/connection-lifecycle.mdx | 7 +-- content/docs/binary-protocol/encodings.mdx | 10 ++--- content/docs/binary-protocol/framing.mdx | 43 ++++++++++--------- content/docs/binary-protocol/index.mdx | 4 +- 6 files changed, 70 insertions(+), 57 deletions(-) diff --git a/content/docs/binary-protocol/cluster.mdx b/content/docs/binary-protocol/cluster.mdx index 954af26ad1..6b814be639 100644 --- a/content/docs/binary-protocol/cluster.mdx +++ b/content/docs/binary-protocol/cluster.mdx @@ -3,15 +3,15 @@ title: Server-to-server description: "The replica-to-replica plane: its dedicated TCP port, command discriminants, and traffic that never reaches a client." --- -Replicas talk to each other with the same 256-byte [framing](/docs/binary-protocol/framing) the client protocol uses, on a **dedicated TCP port**, with their own set of `command` discriminants. None of these frames ever appears on a client connection, and a client cannot reach this plane: it is a separate listener, gated by a handshake. +Replicas talk to each other with the same 256-byte [framing](/docs/binary-protocol/framing) the client protocol uses, on a **dedicated TCP port**, with their own set of `command` discriminants. Client listeners reject replica control commands. The replica listener requires a handshake; peer authentication is optional and must be configured separately. -This page documents the replica plane as of binary protocol 0.11.0. It is internal protocol: only Iggy servers speak it, and it can change between server releases without a client-facing version bump. +This page documents the replica plane used with binary protocol 0.11.0. These internal frames do not negotiate a replica protocol version. Use compatible server builds and follow the [cluster upgrade requirements](/docs/clustering/configuration). ## Transport - Every node in `[[cluster.nodes]]` exposes a `tcp_replica` port next to the client ports: `ports = { tcp = 8090, quic = 8080, http = 3000, websocket = 8092, tcp_replica = 9090 }`. The node's `ip` is the roster address for the replica plane (clients fall back to it when `advertised_address` is unset). - The replica plane is **TCP only**, by design: the prepare hash chain, cross-shard fd delegation, and view-change timing all assume an ordered byte stream. -- **Directional dialing:** a replica dials only peers with a strictly greater `replica_id` and accepts inbound connections only from strictly lower ids. Exactly one connection exists per pair, with no tiebreaker races. +- **Directional dialing:** a replica dials only peers with a strictly greater `replica_id` and accepts inbound connections only from strictly lower ids. This gives each connected pair one dialing direction, avoiding simultaneous dial races. - Frames are the standard `[256-byte header][optional body]` with `size` at offset 48. The maximum frame is **64 MiB** by default (`message_bus.max_message_size`). Headers are `#[repr(C)]`, decoded zero-copy, and require 16-byte alignment. - Nothing inside a frame marks it as replica traffic. The separation is structural: client listeners parse every inbound frame as a `RequestHeader` (command `5` only), the replica listener requires the first frame to be `ReplicaHello`, and the client-bound commands `Reply` (8) and `Eviction` (13) are rejected if they arrive on the replica plane. @@ -23,7 +23,7 @@ The full `command` byte registry, client values included. Values above 29 are re |-------|---------|-----------|---------| | 0 | `Reserved` | - | Invalid sentinel | | 1-4 | `Ping`, `Pong`, `PingClient`, `PongClient` | - | Reserved; no production traffic today | -| 5 | `Request` | client to server | Client command ([framing](/docs/binary-protocol/framing)) | +| 5 | `Request` | client to server; accepted on replica ingress too | Client command; replica ingress decodes the internal routed-request shape | | 6 | `Prepare` | primary to backup, backup to next backup | Replicate one operation | | 7 | `PrepareOk` | backup to primary | Acknowledge a prepare | | 8 | `Reply` | server to client | Client reply; rejected on the replica plane | @@ -55,16 +55,16 @@ There is no dedicated replica heartbeat: liveness is the `Commit` broadcast, sen The `checksum` and `checksum_body` fields that client frames leave zero are live on the replica plane: -- **Frame seal** (`checksum`, bytes 0..16): XxHash3-64 over header bytes 16..256, widened to u128. Sealed on every replica message **except** `Prepare`, `RepairPrepare`, and the three handshake frames (whose integrity comes from the keyed MAC). Verified once, at typed decode, before any field validation; a mismatch drops the frame. -- **Prepare identity** (`Prepare` / `RepairPrepare` only): those two spend `checksum` on an identity hash instead - XxHash3-64 over the whole 256-byte header computed with `checksum = 0` and `view = 0`, so a retransmit that re-stamps `view` keeps the same identity. `PrepareOk` echoes it in `prepare_checksum`, and each prepare's `parent` field carries the previous prepare's identity, forming a hash chain. -- **Body seal** (`checksum_body`, bytes 16..32): three regimes. Metadata-plane prepares seal their body with XxHash3-64. Partition-plane prepares leave it zero: the message batch inside already carries `batch_checksum`, verified at network ingress. `DoViewChange` and `StartView` seal their suffix bodies. Every other message leaves it zero (state-transfer bodies are verified per artifact, not per frame). +- **Frame seal** (`checksum`, bytes 0..16): XxHash3-64 over header bytes 16..256, widened to u128. Sealed on replica control messages. `Request` is unsealed; `Prepare` and `RepairPrepare` use an identity hash instead. Handshake frames use a keyed MAC when authentication is enabled. Verified at typed decode before field validation; a mismatch drops the frame. +- **Prepare identity** (`Prepare` / `RepairPrepare` only): those two spend `checksum` on an identity hash instead - XxHash3-64 over the whole 256-byte header computed with `checksum = 0` and `view = 0`, so a retransmit that re-stamps `view` keeps the same identity. `RepairPrepare` retains the original identity; the receiver restores command `Prepare` before integrity validation. `PrepareOk` echoes it in `prepare_checksum`, and each prepare's `parent` field carries the previous prepare's identity, forming a hash chain. +- **Body seal** (`checksum_body`, bytes 16..32): three regimes. Metadata-plane prepares seal their body with XxHash3-64. Partition-plane prepares leave it zero. `SendMessages` carries its own `batch_checksum`, verified at network ingress; consumer-offset writes have no message batch checksum. `DoViewChange` and `StartView` seal their suffix bodies. Every other message leaves it zero (state-transfer bodies are verified per artifact, not per frame). - The seals are **unkeyed integrity checks, not authentication**. Peer authentication comes from the handshake below; without `cluster.auth` and TLS enabled, the replica port trusts any peer that can reach it. Do not expose `tcp_replica` beyond the cluster network. -`Prepare`, `RepairPrepare`, and the session-forwarding headers validate their reserved regions as zero; the other replica headers leave reserved bytes covered by the frame seal only. +`Prepare` and `RepairPrepare` validate their common and per-command reserved regions as zero. Session-forwarding headers validate their per-command reserved tails. Other reserved bytes in sealed control headers are covered by the frame seal. ## Handshake -Commands 14-16, exchanged before any consensus traffic. These are raw `GenericHeader` frames (`size = 256`) using the per-command area at bytes 128..256: +With `cluster.auth.enabled = true`, commands 14-16 are exchanged before any consensus traffic. With authentication disabled, the dialer sends only `ReplicaHello`. These are raw `GenericHeader` frames (`size = 256`) using the per-command area at bytes 128..256: | Offset | Size | Content | |--------|------|---------| @@ -72,7 +72,7 @@ Commands 14-16, exchanged before any consensus traffic. These are raw `GenericHe | 160 | 32 | BLAKE3 keyed MAC (acceptor's in `Challenge`, dialer's in `Finish`) | | 192 | 1 | `Challenge` only: handshake status | -Statuses: `0` Ok, `1` UnknownCommand, `2` ClusterMismatch, `3` DirectionalRule; values `4` and `5` are reserved (never sent), and a dialer treats any unknown status byte as a rejection. The MAC key derives from `cluster.auth.shared_secret` (32+ bytes; `previous_shared_secret` gives a rotation window), and with cluster TLS enabled (TLS 1.3 only, ALPN `iggy-replica`) the TLS exporter is folded into every MAC as channel binding. The handshake authenticates **cluster membership**, not per-replica identity: one shared secret for the whole cluster. +Statuses: `0` Ok, `1` UnknownCommand, `2` ClusterMismatch, `3` DirectionalRule; `4` AuthRequired and `5` MacMismatch are log-only labels (never sent), and a dialer treats any unknown status byte as a rejection. The MAC key derives from `cluster.auth.shared_secret` (32+ bytes; `previous_shared_secret` gives a rotation window), and with cluster TLS enabled (TLS 1.3 only, ALPN `iggy-replica`) the TLS exporter is folded into every MAC as channel binding. The handshake authenticates **cluster membership**, not per-replica identity: one shared secret for the whole cluster. The `cluster` header field (bytes 32..48) is the first 16 bytes of `blake3(cluster_name)` as a little-endian u128 on every replica frame, checked during the handshake, so nodes from a differently named cluster cannot connect even with auth disabled. @@ -96,7 +96,7 @@ Per-command fields (bytes 128..256): | 224 | 4 | `user_id` | Authenticated user | | 228 | 28 | reserved | Zero | -For metadata operations the body is the admitted command's payload **verbatim**. For `SendMessages` it is the [message batch](/docs/binary-protocol/messages) as stamped by the primary at journal append (`base_offset`, `base_timestamp`, and the recomputed `batch_checksum` are final); each backup re-derives the expected stamp from its own position and refuses a mismatch. Replication is **chain-form**: the primary sends to `(replica + 1) % replica_count`, each backup forwards to its own successor, and the chain stops when the next hop is the primary. `RepairPrepare` (19) is byte-identical to a journaled prepare with only the command byte rewritten. +Metadata bodies contain the server-prepared payload, which may differ from the client request. For example, topic creation adds partition assignments before replication. For `SendMessages` it is the [message batch](/docs/binary-protocol/messages) as stamped by the primary at journal append (`base_offset`, `base_timestamp`, and the recomputed `batch_checksum` are final); each backup re-derives the expected stamp from its own position and refuses a mismatch. Replication is **chain-form**: the primary sends to `(replica + 1) % replica_count`, each backup forwards to its own successor, and disconnected peers are skipped. The chain stops before returning to the primary. `RepairPrepare` (19) is byte-identical to a journaled prepare with only the command byte rewritten. ### PrepareOk (7) @@ -146,17 +146,17 @@ Artifact kinds: | Kind | Artifact | Plane | Content | |------|----------|-------|---------| -| 0 | `METADATA_SNAPSHOT` | metadata | `snapshot.bin` verbatim (MessagePack, snapshot format version 3); `frontier` = sequence number | -| 1 | `CLIENT_TABLE` | metadata | Client table encoding (magic `ICT2`); `frontier` = mutation frontier | +| 0 | `METADATA_SNAPSHOT` | metadata | `snapshot.bin` verbatim (MessagePack, snapshot format version 5; readers accept versions 3 through 5); `frontier` = sequence number | +| 1 | `CLIENT_TABLE` | metadata | Client table encoding (magic `ICT3`, including dedup fences); readers also accept `ICT2` without fences; `frontier` = mutation frontier | | 2 | `SEGMENT_LOG` | partition | One retained segment's `.log` verbatim; `frontier` = segment base offset | -| 3 | `CONSUMER_OFFSETS` | partition | Consumer + group offset tables and applied purge generation; `frontier` = offer's `commit_op` | +| 3 | `CONSUMER_OFFSETS` | partition | `ICO1` version 1: consumer/group offsets, purge generation, next message offset, dedup window, prepare-chain checksum and checkpoint prepare; `frontier` = offer's `commit_op` | ## Session forwarding -A client may dial a backup; authentication happens there, and only the verified identity travels to the primary (credentials never cross the replica plane): +A client may dial a backup; authentication happens there, and only the verified identity travels to the primary (login credentials never cross the replica plane): - **ForwardRegister (26):** `client` 128, `nonce` 144, `user_id` 160. -- **ForwardRegisterResult (27):** `nonce` 128, `client` 144, `epoch` 160, `watermark` 168, outcome byte at **255**: `0` Ok, `1` NotPrimary, `2` NotCaughtUp, `3` PipelineFull, `4` InProgress, `5` Canceled, `6` ClientIdOwnedByAnotherUser. `epoch` and `watermark` must be zero on any non-Ok outcome. +- **ForwardRegisterResult (27):** `nonce` 128, `client` 144, `epoch` 160, `watermark` 168, outcome byte at **255**: `0` Ok, `1` NotPrimary, `2` NotCaughtUp (reserved), `3` PipelineFull, `4` InProgress, `5` Canceled, `6` ClientIdOwnedByAnotherUser. `epoch` and `watermark` must be zero on any non-Ok outcome. - **ForwardLogout (28):** `client` 128, `nonce` 144, `session` 160, `request` 168. - **ForwardLogoutResult (29):** `nonce` 128, `client` 144, `commit` 160, outcome at 255: `0` Ok, `1` NotPrimary, `2` PipelineFull, `3` InProgress, `4` Canceled. @@ -165,6 +165,6 @@ A client may dial a backup; authentication happens there, and only the verified Two consensus planes share this one message set; there are no plane-specific commands. Routing is by the `group: u64` field carried on every consensus header except the handshake and session-forwarding frames (those are implicitly metadata-plane): - `group = 1 << 63` is the **metadata plane** (streams, topics, users, consumer groups; durable on-disk journal). -- Any other value is a packed stream/topic/partition key: a **partition plane** group (message batches, consumer offsets; in-memory journal). +- Any other value is a packed stream/topic/partition key: a **partition plane** group (message batches, consumer offsets). Completion and recovery depend on the independent message and offset [durability policies](/docs/server/durability). -Same wire, a few divergent semantics: partition-plane prepares leave `checksum_body` zero and rely on the batch's own `batch_checksum`, their identity checksum covers the header alone, and `StateTransferTarget.unavailable_transient` / `commit_max` are read only by the partition arm. +Same wire, a few divergent semantics: partition-plane prepares leave `checksum_body` zero, with message integrity supplied by the batch's own checksums; their prepare identity covers the header alone, and `StateTransferTarget.unavailable_transient` / `commit_max` are read only by the partition arm. diff --git a/content/docs/binary-protocol/commands.mdx b/content/docs/binary-protocol/commands.mdx index c97ec88c9d..e70b696fad 100644 --- a/content/docs/binary-protocol/commands.mdx +++ b/content/docs/binary-protocol/commands.mdx @@ -7,11 +7,11 @@ description: "The catalog of command codes, their operation bytes, and which com The u32 command codes. Replicated commands are identified on the wire by their `operation` byte (see [Operation discriminants](/docs/binary-protocol/framing#operation-discriminants)). The code column below is the protocol-level registry and, for non-replicated commands, the value carried in header bytes 196..200. -```bash +```text # System PING = 1 # non-replicated; works without login GET_STATS = 10 # non-replicated -GET_SNAPSHOT = 11 # non-replicated +GET_SNAPSHOT_FILE = 11 # non-replicated GET_CLUSTER_METADATA = 12 # non-replicated DESCRIBE_OPTIONS = 13 # non-replicated GET_ME = 20 # non-replicated @@ -136,7 +136,7 @@ Scope: 1 = topic, 2 = stream, 3 = user. Returns the option catalog for the scope [name_len: u8][name: N][options block to end] ``` -The stream option catalog is empty today, so the block is normally empty (zero bytes). +The stream option catalog is empty, so the block is normally empty (zero bytes). **Delete stream. Code: 203.** @@ -222,7 +222,7 @@ Only `compression_algorithm`, `message_expiry`, and `max_topic_size` are updatab [stream_id: Identifier][topic_id: Identifier][partition_id: u32][segments_count: u32] ``` -Deletes the `segments_count` oldest sealed segments of the partition. +Deletes up to `segments_count` oldest sealed segments of the partition. The active segment is retained. ### Messages @@ -234,7 +234,7 @@ Deletes the `segments_count` oldest sealed segments of the partition. [strategy: 9 bytes][count: u32][auto_commit: u8] ``` -`strategy` is a [polling strategy](/docs/binary-protocol/encodings#polling-strategy). `count` is the requested number of messages. `auto_commit = 1` stores the consumer offset server-side as part of the poll. +`strategy` is a [polling strategy](/docs/binary-protocol/encodings#polling-strategy). `count` is the requested number of messages. `auto_commit = 1` asks the server to advance the consumer offset. A primary submits a quorum offset write before replying, but the poll does not wait for that write to commit. A follower-served poll does not advance the durable offset. See [Client failover](/docs/clustering/client-failover#where-requests-are-served). The response body: @@ -243,7 +243,7 @@ The response body: [batch records to end] ``` -The 16-byte prefix is followed by a stream of [batch records](/docs/binary-protocol/messages) served as stored: each record's header carries the stamped `base_offset` and `base_timestamp`, and each frame's deltas resolve against them. A record may be a server-sliced view of a larger stored batch, so the first polled offset is `base_offset + offset_delta` of the first frame, not necessarily `base_offset` itself. `current_offset` is the partition's newest offset at poll time. +The 16-byte prefix is followed by a stream of [batch records](/docs/binary-protocol/messages): each record's header carries the stamped `base_offset` and `base_timestamp`, and each frame's deltas resolve against them. A record may be a server-sliced view of a larger stored batch, so the first polled offset is `base_offset + offset_delta` of the first frame, not necessarily `base_offset` itself. `current_offset` is the partition's newest offset at poll time. The server rebuilds records when decrypting encrypted payloads. **Send messages. Code: 101.** @@ -256,6 +256,15 @@ The 16-byte prefix is followed by a stream of [batch records](/docs/binary-proto `metadata_length` counts the bytes from `stream_id` through `messages_count` inclusive, so a reader can skip straight to the batch. The producer leaves `partition_id`, `base_offset`, and `base_timestamp` zero in the [batch header](/docs/binary-protocol/messages), and the server stamps them. Every checksum is producer-computed and verified at admission. The reserved regions must be zero. +A committed append returns offset confirmations without a metadata result section: + +```text +[confirmations_count: u32] +[stream_id: u32][topic_id: u32][partition_id: u32][base_offset: u64] x confirmations_count +``` + +Each confirmation is 20 bytes. The server reports one partition per append. A zero count or an empty successful body supplies no offset information; deduplicated requests receive an empty body. Retry coverage is [bounded](/docs/binary-protocol/framing#request-numbering), and crash durability depends on the topic's [durability policy](/docs/server/durability). + **Flush unsaved buffer. Code: 102.** ```text @@ -273,7 +282,7 @@ Parses, but the server always answers `FeatureUnavailable`: there is no on-deman [partition_flag: u8][partition_id: u32] ``` -Response body (20 bytes): +When an offset is stored, the response body is 20 bytes. When none is stored, a successful reply has an empty body. ```text [partition_id: u32][current_offset: u64][stored_offset: u64] @@ -297,7 +306,7 @@ The trailing `ack` byte is mandatory on both write commands. A payload without i | Value | Level | Meaning | |-------|-------|---------| -| 0 | `NoAck` | Leader-local write; respond as soon as local state is updated (the fast path `PollMessages` auto-commit uses) | +| 0 | `NoAck` | Local write on a single-replica partition; replicated partitions use the same consensus path as `Quorum` | | 1 | `Quorum` | Replicate through partition consensus; respond after a quorum commit (default for explicit writes) | ### Consumer groups @@ -364,7 +373,7 @@ Read-only: a member asks for its current partition assignment and the group gene [options block to end] ``` -`permissions_len` and the permissions bytes are present **only** when `has_permissions = 1`. When it is 0 the options block follows immediately. The user option catalog is empty today, so the block is normally empty. `status`: 1 = active, 2 = inactive. +`permissions_len` and the permissions bytes are present **only** when `has_permissions = 1`. When it is 0 the options block follows immediately. The user option catalog is empty, so the block is normally empty. `status`: 1 = active, 2 = inactive. **Delete user. Code: 34.** diff --git a/content/docs/binary-protocol/connection-lifecycle.mdx b/content/docs/binary-protocol/connection-lifecycle.mdx index 8add77d8fe..dff74543c5 100644 --- a/content/docs/binary-protocol/connection-lifecycle.mdx +++ b/content/docs/binary-protocol/connection-lifecycle.mdx @@ -15,11 +15,11 @@ bits 9..0 patch value = major << 20 | minor << 10 | patch ``` -Integer order equals semver order. The value tracks the `iggy_binary_protocol` crate release. Under 0.x the compatibility gate is minor-scoped: the server accepts a client whose packed version is at least the server's minimum and whose `major.minor` is at most the server's. Patch releases never change the wire, so the upper bound ignores patch. Past 1.0.0 the gate follows strict semver (major bump = incompatible). +Integer order equals semver order. The value tracks the `iggy_binary_protocol` crate release. Under 0.x the compatibility gate is minor-scoped: the server accepts a client whose packed version is at least the server's minimum and whose `major.minor` is at most the server's. Patch releases never change the wire, so the upper bound ignores patch. The implemented minimum is the current version with its patch component zeroed. Compatibility across minor versions after 1.0.0 requires a future change to that calculation. ## Login-register -The only way to authenticate is the register handshake: command code **40** (`LOGIN_REGISTER`, username and password) or **45** (`LOGIN_REGISTER_WITH_PAT`, personal access token). Both ride `operation = 1` (`Register`) with `session = 0` and a freshly minted non-zero `client` id. +The only way to authenticate is the register handshake: command code **40** (`LOGIN_REGISTER`, username and password) or **45** (`LOGIN_REGISTER_WITH_PAT`, personal access token). Both ride `operation = 1` (`Register`) with `session = 0`, `request = 0`, and a non-zero `client` id. Mint a fresh id for a new logical client. A client resuming an existing identity must retain its request counter and use the new session epoch returned by registration. Both request bodies begin with the `ClientVersionInfo` prefix, so the server can gate on the version before touching credentials: @@ -52,9 +52,10 @@ After the prefix: An incompatible protocol version is answered with a 256-byte `Eviction` frame, reason `14` (`IncompatibleProtocol`), carrying the accepted window at bytes 144 (max) and 148 (min) as packed u32 versions. A body without a decodable `ClientVersionInfo` prefix gets reason `15` (`MalformedLogin`) with a zero window. Bad credentials get reasons 9-11. See [EvictionHeader](/docs/binary-protocol/framing#evictionheader) for the frame layout. -A successful login is a normal `Reply` whose body is: +A successful login is a normal `Reply` with `status = 0`. Its body includes an empty [result section](/docs/binary-protocol/framing#result-section) before the login payload: ```text +[result_count: u32 = 0] [user_id: u32] [session: u64] [server_protocol_version: u32] diff --git a/content/docs/binary-protocol/encodings.mdx b/content/docs/binary-protocol/encodings.mdx index 515e46f2e0..4fce394d1f 100644 --- a/content/docs/binary-protocol/encodings.mdx +++ b/content/docs/binary-protocol/encodings.mdx @@ -43,7 +43,7 @@ Several commands take an optional partition id encoded as 5 fixed bytes: [flag: u8][partition_id: u32] ``` -`flag = 1` means the id is set. `flag = 0` means none (the value bytes are then zero). For consumer-group consumers the partition id is left unset and the server resolves it. +`flag = 1` means the id is set. `flag = 0` means none (the value bytes are then zero). Group polls and group offset writes require an explicit assigned partition. Plain-consumer polls and consumer-offset reads use partition 0 when the id is omitted. ## Partitioning @@ -57,7 +57,7 @@ How `SendMessages` picks the target partition: - `PartitionId`: `kind = 2`, `length = 4`, value is a u32 partition id. - `MessagesKey`: `kind = 3`, `length = 1..255`, value is a routing key hashed to a partition. -First-party binary SDKs pre-resolve balanced and key routing client-side and normally send `kind = 2`. +The Rust binary SDK resolves balanced and key routing client-side and sends `kind = 2`. SDKs can also send the other wire strategies for server-side resolution. ## Polling strategy @@ -100,7 +100,7 @@ A user headers block (inside a [message frame](/docs/binary-protocol/messages)) [kind: u8][length: u32][data: length bytes] ``` -Fields pair up: first the key, then the value. Keys are `String` kind. Every `length` must be 1..=255, kind 0 is rejected, and the block must consume its byte range exactly. Unknown value kind codes are preserved and forwarded, so headers survive mixed-version clusters. +Fields pair up: first the key, then the value. Keys and values can both use the header kinds above. Every `length` must be 1..=255, kind 0 is rejected, and the block must consume its byte range exactly. The wire validator preserves unknown non-zero kind codes. A typed SDK accessor may still reject a kind it does not recognize. ## Options block @@ -110,7 +110,7 @@ On top of the TLV walk, options enforce: string keys only (kind 2, valid UTF-8), Semantics: -- **Create** requests resolve absent keys to server defaults and persist the effective values, so `GetTopic` always shows what is in force. +- **Create** requests resolve absent keys to server defaults and persist the effective values. An update using a zero retention sentinel can make `GetTopic` option values disagree with its fixed expiry/size fields; see [update semantics](/docs/server/topic-options). - **Update** requests are patches: keys absent from the block are left alone, never reset. A client built before a key existed cannot erase it. - Unknown keys are rejected at the wire edge, never silently skipped. @@ -125,4 +125,4 @@ The `compression_algorithm` value used by topic options: | 1, `none` | No compression (default) | | 2, `gzip` | Gzip | -Any other value is rejected. The option is a **placeholder today**: the value is validated, persisted, and echoed back, but neither the server nor the SDKs compress or decompress payloads yet. To compress today, do it client-side and tag messages via user headers - see the [message headers examples](https://github.com/apache/iggy/tree/master/examples/rust/src/message-headers) in the Iggy repo. +Any other value is rejected. The option is a **placeholder today**: the value is validated, persisted, and echoed back, but it does not enable automatic payload compression or decompression. To compress today, do it client-side and tag messages via user headers - see the [message headers examples](https://github.com/apache/iggy/tree/master/examples/rust/src/message-headers) in the Iggy repo. diff --git a/content/docs/binary-protocol/framing.mdx b/content/docs/binary-protocol/framing.mdx index e3bf5148a3..896b11d3fd 100644 --- a/content/docs/binary-protocol/framing.mdx +++ b/content/docs/binary-protocol/framing.mdx @@ -36,9 +36,9 @@ Client to server. 256 bytes. | 60 | 1 | `command` | u8 | `5` (`Request`). | | 61 | 1 | `replica` | u8 | Send zero. | | 62 | 66 | reserved | bytes | Zero. | -| 128 | 16 | `client` | u128 | Client-chosen session identity, non-zero. Minted fresh for each registration. | -| 144 | 16 | `request_checksum` | u128 | Optional integrity stamp over the request body (the Rust SDK uses XxHash3-64 widened to u128). Lets the server's client table catch a `request` number reused for different arguments. Zero disables the comparison. Stamped only for metadata-plane operations; zero for partition-plane and non-replicated ones. | -| 160 | 8 | `timestamp` | u64 | Informational; the server echoes it into `ReplyHeader.timestamp`. May be zero. | +| 128 | 16 | `client` | u128 | Client-chosen identity, non-zero. Mint a fresh id for a new logical client; retain it when resuming that client. | +| 144 | 16 | `request_checksum` | u128 | Optional integrity stamp over the request body (the Rust SDK uses XxHash3-64 widened to u128). Lets the server's client table catch a `request` number reused for different arguments. Zero disables the comparison. The Rust SDK stamps metadata/session operations and `DeleteSegments`; it sends zero for partition operations and non-replicated ones. | +| 160 | 8 | `timestamp` | u64 | May be zero. Direct replies can echo it, but committed prepares use a server-assigned timestamp. | | 168 | 8 | `request` | u64 | Request number, per client. See [request numbering](#request-numbering). | | 176 | 1 | `operation` | u8 | The [`Operation`](#operation-discriminants) discriminant. | | 177 | 7 | padding | bytes | Zero. | @@ -87,19 +87,22 @@ Value 0 is reserved and rejected. The 64..127 range is reserved for server-inter The planes matter for delivery semantics: -- **Metadata operations** replicate through the metadata consensus group. The server deduplicates them by `(client, request)` and caches replies, so a retried request gets the cached answer instead of a double apply (exactly-once). -- **Partition operations** replicate through their partition's consensus group. They are at-least-once: no reply cache, a replay may apply again. -- **Non-replicated operations** are reads. They bypass consensus and deduplication entirely. +- **Metadata operations** replicate through the metadata consensus group. The server deduplicates them by `(client, request)` within its retained client history. Recent replies are cached and replayed; an older request can remain recognized as already applied after its reply has left the cache. This is not an unlimited exactly-once guarantee. +- **Partition operations** normally replicate through their partition's consensus group. Each partition retains client request watermarks and a 128-id committed window to suppress duplicates. There is no reply cache: a recognized duplicate gets an empty success, including no offset confirmations for a retried send. Capacity eviction or a new client identity loses this coverage. Single-replica consumer-offset writes with `ack = 0` bypass consensus and this deduplication. +- **Non-replicated operations** bypass consensus and deduplication. A poll can still submit a separate consumer-offset write when auto-commit is enabled. ## Request numbering `request` is a per-client counter the server's client table tracks for metadata-plane operations: -- Metadata operations must send a strictly increasing `request` (the SDK advances the counter per metadata request). -- Partition operations and non-replicated operations send the current counter value without advancing it. The server doesn't track theirs. +- New metadata operations must use a `request` above the retained watermark; a retry reuses its original number and arguments. +- The Rust SDK advances one shared counter for metadata operations, partition operations, logout, and `DeleteSegments`. Non-replicated operations use the current value without advancing it. +- Register uses `request = 0`; other operations except `NonReplicated` require a non-zero request number. - The counter is a watermark, not a contiguous sequence: any value above the last accepted one is admissible. -`session` is the fence epoch: the value handed back by the login reply. Every request after login must echo it. When the same `client` id registers again, the new registration mints a higher epoch and requests carrying the old one are fenced (rejected as zombies). +`session` is the fence epoch handed back by the login reply, derived from the committed Register log position. Echo it after login. Registering the same `client` again mints a higher epoch; the metadata client table rejects an older epoch. Partition dispatch uses the identity bound to the authenticated connection. + +The partition window is measured in the shared client request-id space. An unseen request arriving 128 or more ids below that partition's watermark is treated as already committed and receives success without executing. Avoid allowing an unresolved write to fall outside that window; the protocol does not provide an unlimited reordered-retry guarantee. ## ReplyHeader @@ -111,21 +114,21 @@ Server to client. 256 bytes, followed by `size - 256` bytes of body. | 16 | 16 | `checksum_body` | u128 | Zero on client-facing frames. | | 32 | 16 | `cluster` | u128 | Cluster id. | | 48 | 4 | `size` | u32 | Total frame length: 256 + body length. | -| 52 | 4 | `view` | u32 | Consensus view the reply was produced in. | +| 52 | 4 | `view` | u32 | View associated with the reply. A cached committed reply retains its original view; direct replies can echo the request value. | | 56 | 4 | `release` | u32 | Zero. | | 60 | 1 | `command` | u8 | `8` (`Reply`). | -| 61 | 1 | `replica` | u8 | Answering replica index. | +| 61 | 1 | `replica` | u8 | Replica index stamped by the reply builder. Cached committed replies retain the original primary index. | | 62 | 66 | reserved | bytes | Zero. | | 128 | 16 | `request_checksum` | u128 | Echoed from the request. | | 144 | 16 | `context` | u128 | Server context. | | 160 | 16 | `client` | u128 | Echoed client id. | -| 176 | 8 | `op` | u64 | Log position of the committed operation. | -| 184 | 8 | `commit` | u64 | Commit point at reply time. | -| 192 | 8 | `timestamp` | u64 | Echo of the request `timestamp`. | -| 200 | 8 | `request` | u64 | Echoed request number; correlate replies by this. | -| 208 | 1 | `operation` | u8 | Echoed operation discriminant. | +| 176 | 8 | `op` | u64 | Committed log position on consensus replies. Direct replies can instead carry the session epoch or zero; this field alone does not prove a commit. | +| 184 | 8 | `commit` | u64 | Commit position associated with the reply. A cached committed reply retains its original position. | +| 192 | 8 | `timestamp` | u64 | Server-assigned prepare timestamp on committed replies, or the request timestamp on direct replies. | +| 200 | 8 | `request` | u64 | Echoed request number. Non-replicated requests can reuse a number, so it is not a unique correlation id for concurrent reads. | +| 208 | 1 | `operation` | u8 | Reply operation. Server rewrites can return `64` for CreateTopic, `65` for CreatePartitions, or `68` for DeleteSegments instead of the original client operation. | | 209 | 7 | padding | bytes | Zero. | -| 216 | 4 | `status` | u32 | `0` = accepted. Nonzero = an `IggyError` code for a failure decided **before** commit (authorization denial, admission reject). A nonzero status always comes with an empty body. | +| 216 | 4 | `status` | u32 | `0` = no header-level failure. Nonzero = an `IggyError` code for a failure decided **before** commit (authorization denial, admission reject). A nonzero status always comes with an empty body. | | 220 | 36 | reserved | bytes | Zero. | Decode order for a client: @@ -137,7 +140,7 @@ Decode order for a client: ## Result section -Replies for all metadata operations and for the partition-plane consumer-offset writes (`StoreConsumerOffset`, `DeleteConsumerOffset`) are **result-framed**: the body starts with a committed-result section ahead of the typed payload. +Replies for all metadata operations, including their server-internal rewrites, and for the partition-plane consumer-offset writes (`StoreConsumerOffset`, `DeleteConsumerOffset`) are **result-framed**. Non-empty Register replies use the same framing. The body starts with a result section ahead of the typed payload. ```text [count: u32] @@ -145,9 +148,9 @@ count x { index: u32, result: u32 } ``` - Success: `count = 0`, and the typed response payload (if any) follows the 4 count bytes. -- Committed business rejection: one entry `{ index: 0, result: error_code }` and no payload. +- Business or transient rejection: one entry `{ index: 0, result: error_code }` and no payload. -The header `status` channel and the result section are mutually exclusive by construction: a reply either failed pre-commit (`status` nonzero, empty body) or committed (`status` zero, result section present). A login-register reply carries the result section only when non-empty. On success its body starts directly with the [login response payload](/docs/binary-protocol/connection-lifecycle#login-register). +A nonzero header `status` has an empty body. With `status = 0`, decode the result section for the operations above: it can report either a committed outcome or a pre-commit transient rejection. Its presence alone does not prove commitment. A successful login-register reply starts with `count = 0`, followed by the [login response payload](/docs/binary-protocol/connection-lifecycle#login-register). An empty Register body is a terminal failure, not a successful login. Replies to non-replicated commands aren't result-framed: after `status = 0` the body is the response payload directly. diff --git a/content/docs/binary-protocol/index.mdx b/content/docs/binary-protocol/index.mdx index a97d18a3d9..0040cfff30 100644 --- a/content/docs/binary-protocol/index.mdx +++ b/content/docs/binary-protocol/index.mdx @@ -3,9 +3,9 @@ title: Binary Protocol description: "Binary protocol 0.11.0: one wire format across TCP, QUIC and WebSocket, covering both the client and replica planes." --- -Iggy speaks one binary protocol over its three binary transports: TCP, QUIC, and WebSocket. Commands, responses, data models, and status codes are the same on all of them. The HTTP transport is separate: it exposes the same operations as JSON REST endpoints, listed in [server.http](https://github.com/apache/iggy/blob/master/core/server/server.http). +Iggy speaks one binary protocol over its three binary transports: TCP, QUIC, and WebSocket. Commands, responses, data models, and status codes are the same on all of them. The HTTP transport is separate: it exposes JSON REST endpoints for a subset of these operations, listed in [server.http](https://github.com/apache/iggy/blob/master/core/server/server.http). -This section describes **binary protocol version 0.11.0**. The protocol version is the semver of the `iggy_binary_protocol` crate, and it's exchanged and checked during login (see [Connection lifecycle](/docs/binary-protocol/connection-lifecycle)). The crate itself is the source of truth: every request and response module carries a `Wire format:` doc comment, and these pages are written against those. +This section describes **binary protocol version 0.11.0**. The protocol version is the `major.minor.patch` of the `iggy_binary_protocol` crate, ignoring prerelease tags, and it's exchanged and checked during login (see [Connection lifecycle](/docs/binary-protocol/connection-lifecycle)). The crate itself is the source of truth: request and response types document their payload layouts, and the server dispatch defines which operations are supported. All multi-byte integers are **little-endian** unless stated otherwise. From 2d570fa75d1df3540adf90cf2fa7ad9132913776 Mon Sep 17 00:00:00 2001 From: hubcio Date: Fri, 11 Sep 2026 04:24:43 +0200 Subject: [PATCH 06/13] fix(docs): align MCP setup and permissions --- content/docs/ai/mcp.mdx | 51 +++++++++++++++++++++++++++++++---------- 1 file changed, 39 insertions(+), 12 deletions(-) diff --git a/content/docs/ai/mcp.mdx b/content/docs/ai/mcp.mdx index 8e8e716b0b..201150e7ef 100644 --- a/content/docs/ai/mcp.mdx +++ b/content/docs/ai/mcp.mdx @@ -7,9 +7,15 @@ The [Model Context Protocol](https://modelcontextprotocol.io) (MCP) is an open p ## Getting started -To start the MCP server, simply run `cargo run --bin iggy-mcp`. +Start an Iggy 0.9.0 or edge broker using the [getting started guide](/docs/introduction/getting-started). From the matching `apache/iggy` checkout, run: -The [docker image](https://hub.docker.com/r/apache/iggy-mcp) is available, and can be fetched via `docker pull apache/iggy-mcp`. +```bash +IGGY_MCP_IGGY_USERNAME=iggy IGGY_MCP_IGGY_PASSWORD=iggy cargo run --bin iggy-mcp +``` + +These are the development credentials configured in that guide. Use your broker's credentials or a PAT when connecting to an existing installation. + +The [docker image](https://hub.docker.com/r/apache/iggy-mcp) is available, and can be fetched via `docker pull apache/iggy-mcp:edge`. ## Configuration @@ -68,13 +74,15 @@ transport = "grpc" # grpc or http endpoint = "http://localhost:4317" ``` -The configuration file must be in the `toml` format. The path to the configuration can be overridden by `IGGY_MCP_CONFIG_PATH` environment variable. Each setting can also be overridden by using the following convention `IGGY_MCP_
_` e.g. `IGGY_MCP_IGGY_USERNAME`, `IGGY_MCP_HTTP_ADDRESS` and so on. Environment variables can also be loaded from a dotenv file: point `IGGY_MCP_ENV_PATH` at the file, otherwise a `.env` file in the current working directory is loaded automatically. +The configuration file must be in the `toml` format. By default, the server looks for `core/ai/mcp/config.toml` relative to its working directory. Set `IGGY_MCP_CONFIG_PATH` to use another path. Embedded defaults are loaded first, then the file if it exists, then environment overrides. + +Each setting can also be overridden using `IGGY_MCP_
_`, for example `IGGY_MCP_IGGY_USERNAME` or `IGGY_MCP_HTTP_ADDRESS`. Nested settings use the same underscore convention, such as `IGGY_MCP_IGGY_TLS_ENABLED`. Set `IGGY_MCP_ENV_PATH` to load a particular dotenv file; otherwise `.env` is searched for in the current directory and its parents. Existing environment variables take precedence over dotenv values. -The `token` value can be either a literal PAT or a `file:` reference such as `token = "file:/run/secrets/iggy_pat"`, in which case the token is read from the given file (`~` is expanded to the home directory). +The `token` value can be either a literal PAT or a `file:` reference such as `token = "file:/run/secrets/iggy_pat"`, in which case the token is read from the given file and surrounding whitespace is removed (`~/` is expanded to the home directory). A non-empty token takes precedence over the username and password. ## Available tools -The MCP server exposes 40+ tools covering the full Iggy API: +The MCP server exposes these 41 tools: ### Server @@ -169,11 +177,13 @@ update = false delete = false ``` +`poll_messages` requires `update` as well as `read` when `auto_commit` is true or the `next` strategy is used. The `next` strategy enables auto-commit automatically. Storing an offset requires `update`; deleting an offset requires `delete`. PAT creation and deletion require `create` and `delete`, respectively. + On top of this, the Iggy user account used by the MCP server has its own granular permissions. For production use, create a dedicated user with the minimum required permissions. ## Claude Desktop integration -Here's the example configuration to be used with Claude Desktop: +Set `command` to the absolute path of the built `iggy-mcp` executable. This Claude Desktop example uses the broker and development credentials from the getting started guide: ```json { @@ -182,7 +192,10 @@ Here's the example configuration to be used with Claude Desktop: "command": "/path/to/iggy-mcp", "args": [], "env": { - "IGGY_MCP_TRANSPORT": "stdio" + "IGGY_MCP_TRANSPORT": "stdio", + "IGGY_MCP_IGGY_ADDRESS": "localhost:8090", + "IGGY_MCP_IGGY_USERNAME": "iggy", + "IGGY_MCP_IGGY_PASSWORD": "iggy" } } } @@ -191,18 +204,32 @@ Here's the example configuration to be used with Claude Desktop: ## Docker -Run the MCP server as a container: +Create a shared network and start a development broker that advertises its container name: ```bash -docker run -e IGGY_MCP_TRANSPORT=http \ +docker network create iggy-mcp +docker run -d --name iggy-server --network iggy-mcp \ + --security-opt seccomp=unconfined --ulimit memlock=-1:-1 \ + -e IGGY_ROOT_USERNAME=iggy -e IGGY_ROOT_PASSWORD=iggy \ + -e IGGY_TCP_ADDRESS=0.0.0.0:8090 \ + -e IGGY_NODE_ADVERTISED_ADDRESS=iggy-server \ + apache/iggy:edge +``` + +After the broker is ready, run the MCP server on the same network: + +```bash +docker run --rm --network iggy-mcp -e IGGY_MCP_TRANSPORT=http \ -e IGGY_MCP_HTTP_ADDRESS=0.0.0.0:8082 \ -e IGGY_MCP_IGGY_ADDRESS=iggy-server:8090 \ -e IGGY_MCP_IGGY_USERNAME=iggy \ -e IGGY_MCP_IGGY_PASSWORD=iggy \ - -p 8082:8082 \ - apache/iggy-mcp + -p 127.0.0.1:8082:8082 \ + apache/iggy-mcp:edge ``` +The HTTP endpoint uses the configured Iggy account for every MCP client. This example publishes it only on the host loopback interface. + The default HTTP address is `127.0.0.1:8082`, which inside a container is unreachable from the outside, so `IGGY_MCP_HTTP_ADDRESS=0.0.0.0:8082` is required for the published port to work. ## Systemd integration @@ -213,4 +240,4 @@ Build with the `systemd` cargo feature to enable systemd readiness and watchdog cargo build --bin iggy-mcp --release --features iggy-mcp/systemd ``` -The MCP server then behaves the same way the Iggy server does under systemd. +Readiness is sent after the HTTP listener starts or the stdio MCP session initializes. When systemd enables the watchdog, the server sends keep-alive notifications at half the configured watchdog interval. SIGINT, SIGTERM, or stdio client disconnect stops the server and sends a stopping notification. From 7e3a1adf018e3b7a2eb2e82b47b554461bb5886c Mon Sep 17 00:00:00 2001 From: hubcio Date: Fri, 11 Sep 2026 05:30:51 +0200 Subject: [PATCH 07/13] fix(docs): align Web UI setup and features --- content/docs/introduction/about.mdx | 4 +-- content/docs/server/configuration.mdx | 2 +- content/docs/web_ui/start.mdx | 34 ++++++++++++++++++-------- public/img/iggy_web_ui.png | Bin 145001 -> 49250 bytes 4 files changed, 27 insertions(+), 13 deletions(-) diff --git a/content/docs/introduction/about.mdx b/content/docs/introduction/about.mdx index 9c8129b438..f2d6b620f6 100644 --- a/content/docs/introduction/about.mdx +++ b/content/docs/introduction/about.mdx @@ -81,10 +81,10 @@ Iggy CLI can be installed with `cargo install iggy-cli --version 0.14.0-edge.7 - The Web UI provides a comprehensive dashboard for the Iggy server, built with SvelteKit and TypeScript. It can run in two modes: -- **Embedded** - compiled into the server binary (with `iggy-web` feature flag), served at the `/ui` endpoint when `http.web_ui = true` +- **Embedded** - compiled into the server binary (with the `iggy-web` feature and [built static assets](/docs/web_ui/start)), served at the `/ui` endpoint when `http.web_ui = true` - **Standalone** - as a separate container via `docker pull apache/iggy-web-ui:edge` -Features include stream/topic/partition management, a message browser with JSON/string/XML decoders, and user management. The logs and terminal pages are placeholders, and the server settings page has a disabled Save action rather than an operational configuration editor. +Features include stream/topic/partition management, a message browser with JSON/string/XML decoders, and user listing and creation with initial permissions. Editing or deleting existing users and changing their permissions are placeholders. The logs and terminal pages are also placeholders, and the server settings page has a disabled Save action rather than an operational configuration editor. ## Connectors diff --git a/content/docs/server/configuration.mdx b/content/docs/server/configuration.mdx index a550393b1e..fb2340bd3b 100644 --- a/content/docs/server/configuration.mdx +++ b/content/docs/server/configuration.mdx @@ -115,7 +115,7 @@ The tables below list every section with its shipped defaults. In cluster mode, HTTP sessions hold live server-side session state (they count against `metadata.clients_table_max`). Consumer group management (create, get, delete) is available over HTTP. Group membership (join, leave) is not and needs a stateful transport. -In cluster mode, followers forward control-plane requests (streams, topics, users, and so on) to the current primary when a cluster-wide JWT key exists (see `[http.jwt]` below). Forwarding does not cover the partition plane: message produce and consumer-offset writes must reach the partition's primary node directly, while message polls read locally on any node. +In cluster mode, followers forward control-plane requests (streams, topics, users, and so on) to the current primary when a cluster-wide JWT key exists (see `[http.jwt]` below). Message produce and consumer-offset writes use a separate fallback: after a `TransientNotAccepted` response, the server tries each other roster node at most once. Message polls read locally on the receiving node. ### `[http.cors]` diff --git a/content/docs/web_ui/start.mdx b/content/docs/web_ui/start.mdx index e1ebd9f958..9d5e23afab 100644 --- a/content/docs/web_ui/start.mdx +++ b/content/docs/web_ui/start.mdx @@ -7,6 +7,8 @@ Iggy Web UI provides a comprehensive dashboard for Iggy server. It allows you to ![Web UI](/img/iggy_web_ui.png) +These instructions target Iggy server 0.9.0. The container examples use `edge` while the release is being prepared. Use the matching source checkout for local builds. + ## How it connects The Web UI is a purely client-side application: every call to the Iggy HTTP API is issued by your browser, not by the Web UI container or server. Two things follow from this: @@ -28,22 +30,32 @@ enabled = true web_ui = true # Enable embedded Web UI at /ui ``` +When building from source, generate the static assets before compiling the server. Run from the repository root: + +```bash +npm --prefix web ci +npm --prefix web run build:static +cargo build --bin iggy-server +``` + +The feature alone does not generate these assets. A server binary built without them returns 404 at `/ui`. + If the server is compiled without the `iggy-web` feature and `web_ui = true`, a warning is logged but the server continues to run normally. ### Standalone mode -Run the Web UI as a separate container: +Start an Iggy server with HTTP enabled, as shown in [Getting started](/docs/introduction/getting-started), then run the Web UI as a separate container: ```bash -docker pull apache/iggy-web-ui -docker run -e PUBLIC_IGGY_API_URL=http://localhost:3000 -p 3050:3050 apache/iggy-web-ui +docker pull apache/iggy-web-ui:edge +docker run -e PUBLIC_IGGY_API_URL=http://localhost:3000 -p 3050:3050 apache/iggy-web-ui:edge ``` `PUBLIC_IGGY_API_URL` is consumed by the browser: if the Iggy server also runs in Docker, use the host-published HTTP port here, not the container's network alias. ### Local development -The Web UI lives in the `web/` folder of the [repository](https://github.com/apache/iggy/tree/master/web). Use a Node.js version supported by the frontend toolchain (Vite requires `^20.19.0 || >=22.12.0`, so 22 LTS is a safe default) and `npm` (`pnpm` and `yarn` are not part of the supported workflow): +The Web UI lives in the `web/` folder of the [repository](https://github.com/apache/iggy/tree/master/web). Use a Node.js version supported by the frontend toolchain (`^20.19.0 || ^22.13.0 || >=24`, for example Node.js 22.13 or later in the 22.x series) and `npm` (`pnpm` and `yarn` are not part of the supported workflow): ```bash cd web @@ -59,16 +71,18 @@ The Web UI provides the following pages and functionality: - **Overview** - server health and key metrics - **Streams** - list, create, and manage streams -- **Topics** - manage topics per stream (create, update, delete, purge) +- **Topics** - create and delete topics, and edit their name and message expiry - **Partitions** - view partition details per topic - **Messages** - browse messages per partition with built-in decoders: - JSON decoder (formatted JSON output) - String decoder (UTF-8 text) - XML decoder -- **Users** - user management (create, update, delete, permissions) +- **Users** - list and create users, including their initial permissions - **Settings** - Web UI preferences -Some pages visible in the navigation (Clients, Logs, terminal, server settings) are placeholders that aren't functional yet. See the [roadmap](https://github.com/apache/iggy/tree/master/web#roadmap) for what is planned. +Topic updates share the HTTP API's [retention update limitations](/docs/server/topic-options). Other topic options require the CLI or HTTP API. + +The Clients, Logs and terminal routes are placeholders hidden from navigation. The Server settings page displays information, but saving settings is disabled. Editing and deleting existing users, changing their permissions and bulk user actions are also placeholders. See the [roadmap](https://github.com/apache/iggy/tree/master/web#roadmap) for what is planned. ## Docker Compose example @@ -77,7 +91,7 @@ Here's the full example of the `docker-compose.yml` file that starts the Iggy se ```yaml services: iggy: - image: apache/iggy:latest + image: apache/iggy:edge container_name: iggy restart: unless-stopped cap_add: @@ -111,7 +125,7 @@ services: - iggy:/app/local_data init-iggy: - image: apache/iggy:latest + image: apache/iggy:edge container_name: init-iggy networks: - iggy @@ -130,7 +144,7 @@ services: " iggy-web-ui: - image: apache/iggy-web-ui:latest + image: apache/iggy-web-ui:edge container_name: iggy-web-ui restart: unless-stopped environment: diff --git a/public/img/iggy_web_ui.png b/public/img/iggy_web_ui.png index 1fa0d7db6c616881087d12d683360ce77f0492e5..39350b7371759ded76cd7d72745d1bfbe17cf532 100644 GIT binary patch literal 49250 zcmce;WmuHmA2o^(B2uCPBH(~Zmq<4V2uOFAbcb}Kbcb~J(A^Cp-Q6V(!%#!l*-!k> z^?o>C&pYs8F0O%l-}nB-`mMD#L9)`KSm=c4NJvOnVqZSXBO#$=AR!^|JpBv&B#&S? z2MOsplGtYfMduXQ5{fFW7SWScIYITt928tMypKo~KIO7Gca-`iEYryT$O5b68jyVd{RBDDs5NB-~qCGlU+{(HOX_W$39y)pm0$R`4{r%#Vg z8ed&s+1X37EGfw;kBv^kCi}W)XA`JpSpU5I`FgciQYJns*6@GpV$aRXBbjZpzB#u{ z=l6m(+1j2E67I1)CnSuEd@h4LKR^FLUlSF}_7x6}e*dGt!XMtQ{`aew$oLqCiS zQ&SIMJoO5cL>Lkh)YR1Q$(R^i9g=ifl@BC5&Y$GvZ5L`Gj_2o+YC`SD>1ISW?%-Pe z8)B?xld4sYkR_)7{$6ga`HXa-;{5zviB6lGl2VN?XXme96jDhxO9STFH_#5BfRBdg z9yW?p!NoiuoPU27?dj^;+}<7-7^pK_Y7`E6b+$KUOoNt7ghLs+QfJM8wY|A%IlB`c z`rm$t?_O49$5_>eL+UK%PIQzhr>3T4(s+3FyAj9h-N8XYXYi!}YHkkib^gt7k9#OC z!dT6xbLb)X_}q5gp$d7@<63xlcvD;se9d0zF$j-M4YiutW<&F(Mu)^;;oa_?G13SK zZ$D(BPN(f)!Rh__zc=<>2mR?&laoAQlLRP*0r{F2ah_>sR&F4Q+SRY;Me74lIxw$Em#9|tG`W2q|<;Ra7Dy5oy zu87{A)m8NxGs~sgrLhWj1-bg&Cq!nWua}#8|BK9w$1W}|TU*d1!ldnCH(3onCwE83 z^U7QaHcQ))k&(P6Z*OlL3dt1K1u0e41a54pz4=@_clYMHh_dhBkrZC_Gp>G-lDfL( zWML&2{-T;MfGmWD7W8vFPYH*Zm>7a5e5g<(4Y9PegzCjAw7<0BYQTgZU3O@qQ=*^rkeJTmZAW4fhma( zznMw?KwoC3`ah5EGX7w;!lN5uI)eOHO?9<%PsD4++q$6@ELRq{0bT!FL)4g6`Ky!bkY%U^!(oqOl zJ~^3^No5hBmu!z`WPD&IRoE)qKWETxA^5`;y(T1v78Ii?E-_!Sg5r;Sn>LlX}^Ad=N8Y)kjmxJ-94==EuD66c>t46<#W4V`-D+vhxo zS9vV3K|^m{+U1H72*eNM>o5lgmwOn&;U&1sG+w*LScee(p72+$y;HO*^EH+pm!krO zM5o~ow^yezebJ3h2Z94RO(Cxs@Ls;m`cY@SB74-qP04$1`1LK;)2C1AG+Mn2#LYgU z%n6jJ*Tm6jTNs;H8JU*6m! zF=99?ZEig+ZtIQsDDMdf2r#$F!p|XG(g_?DO+wh5oScsSq7k=obn0J_xwyDUMZ)z& zKd3cR@zRGupg#;H;>$ZbMMHDu5(;yKdcqwX9Fkc~`u?@?b6BkQLl%`vZTRZoY%n>H;l%RM1W_GS6_?gv6UJS&U9*6p zVv*%nFPSDP^8)c9f%^DhBJY#BNaJh)G4XaTSyuoyi3svEJ)M$LOj1I!L@Kp&Tc7uv3OV+CM3 zLK(t~i;IJBx=HvQL!ph0jibo)N$l&bp<+0cvi=+vJI`Hv@9%C*+$AMLPcwczIXj!$ z%OB19;O1(JFU?sLGLZjD`4vk+W>D1r>-f>pQGREIe&#QD#|XJzH#{zBo*WAJ+U zvkM{nj-DPuM5JM2P^4Uvo0F6OXC%h2vRRUD$_&2PFyzkvQ*SdeQ>fu=JDx#@%YOf7 zN=iy{vV_a;R%8^^LC`J2-Y|NncKjN*oq@NAi;Kgw3X`SMSk@2)3M|b|hiP&j^s8%X z@}!bI5xr{$;jy$TblR0M&{Fl9pUNbiYHFbF=oKd5(JGI7Q5##$w6>I&AEh=qQQ_(J z3+y({er0|}_?4ktzW<(siHWJIs*06mGI6}GcTl~`B?8(gzLJ)Tf`XFP`tqNTxwR2b!|-Z#gOmTkgW6jh^<*8zU= zvqmGnUP@1AuPEc+9ZudPko8N_2elC}wNWlWlolMocVg)gCO=agu)b-0Xvy$B-$5kt&V zFOz=djjG)uUMC_V0);~T;u<_|YIi4~Oeh|$3I2Y5>6~`+jfN6JLfJCud~&OT>qIOc zbFft)HC&4UqwHVG(s*}JM$5Cj-?yA7(`$Sd5eW_sE}Sk9k`aFpjes03Hb`{ryI;ip zg?k-nYGN{KocQWy>mWIq-TDEZtoKjUtm?dYMQmhZqFP<}g_OjVpN{Jx;NsBI!U7{K z2hpA(;B$0j1Eq_Zd+|mwV!lG3o14eHc41Vv(|Lt*s(-pztLK?clOroLb2Rl%;kv52 zIt^BSK|v~qOH#7c^FqT0>-k@fbz)PtBNF(gp9JM?;mubt%qBI`DcAV_#!@k&Xku`^du&>e6`y)j}IGoPtI2qacK?ozf9Nksrqn&}G0A5S3 zVKfvm6Bs=I2yUU^+f7PhPe#IjQJ&Gfd*mZ#8#vCs`F58p>=ua>MV^ zPl11NxRR&A>*0ZcAwgJwvL2Z)fzL2%dAKYs9PhI2C@?+R6fu$7bYa?8XSKAu;)Udq z!Sg}Wz}UtCBGIW)XSK~o_VMFKUS7-1ekFVc7dDGYle0ag7%N$d6Rw@1iM)|rQpwX| ztvL}9-6HG9v6_1?g?}=Gbf*_+cyhEq>ahR z1zL@F-F}?Q&O~vtXKt!FBg(PNYAB9Lbp_aY>2dyT#sles7 zloTGfY2+H_s%Vq@Ec`Ce9@VR??f{0{+}t31O;|~a`pkB@K_8X%g+iBe_yV0e<^qOk zi|bh`Dzwa{7w6|jJ)`~{E+@Z1Ki1<|2rvB2 zR$JTRcA;Ce!^hWJtu;6}n42R$WoTq3URXV7ZC_fTEvhCbFnv0I@cXwxKZhjd{;{@d zK9vh+BK-WI!HK#&pPbG@RVSjQNJAyWN{BC@ioiGj3q3p~82!XP`RC7WNOlfY8MN@% zvzdu0&xdz84ObWD3`CQ(90Szk4R2k|O*9{76D5dh53OhV$LD`(iWcR-sHrPfy+1Qm zG7Xl6Lb#>n>xCur@MrJ^KAXcVJKOuqP;mS%%cT4Oj|LpC9f7wdw*2WetMCPm? z1p_~ffZ26qc}bDCZ(x91Knc(50gGsf8jFgHe<(6GwkyVu34=pWkgozcOiC&Qis!p$ zq~8;^4LRl9VCI%ZyTgcVXQq-^i|`rLzf37YQX3@&veu*u6zmldzCk#$U z3vx2hJGoH!0(awYy*c{kY+!FIZZ;+?CLU-P;D>_2RaT;PbF;=Tn9_t$%hf;^P^M4u zygvz}ojDx$Vm|fnL_OQ`pexIX!PV8(?bFeEmS}RnW1uQoylR546niKak{|fAvoCYD zsl}CZIEj;#g#~eSy3KB-Y0N>%IM#jA=vbmscG!`Zlr+DeQCGFJMMXtZs#V2x*%=^= zfe%}%a$9Ld9!g?oxH4p6vmAsNai1NZd+rR`(GhqIkWo@fh>2O+I<-BlE(6RV=$UeQ z((}Kp#5%D(DbsGHa~$uD%HErv(yF!S^S(OGmseY9Uv2L3tF>IfO`j0vhh17k?> z^bql1?KaB>nPQX98aq2N(4jAO=eM};H8u*fL?hm8@uCI%iYicaGX##^nGGtdRdBzcvmH&V&dY6JuCBx%&7ynR>~0_ z?H1Fzn|_yE_hR3ARU_r|97Z1((aIJJXU$d@7(%$n2>MFOa&yyw-`_WlW517EBP=O5 ztW!}brWMyFN~=SIgY{0QC+Kx>VovvZS@y!k z?>=tyvBO~$4lglk`v~;Uqq{Zaq zls40bKOvj#?H!{X4_aTJA83e+CuL>voQhHv?&vMBvAjQNt@DzWE-f#Q%=rFxLEjJ! z6AK|!y7cNrX>MMTlF9gK``bnbg6W96oEDdhA6MKo;^^4{=N@>75AT_vvjjhX{@hho z-eQ0Dl{(Dx?na^g+h+eJ7;gy(60MdRCwjwExpJX~N5hV%TT+1KKtbu^^GNaLcUP^@ zb1~Yw5NbA?DNX(P^TdB5kt#Q`nRn?Z3ldB0e0h`!`a?yFk*}Yhux|i+UHMa2g>t6- z$pYrEQ-T~!zkkNK?`CCU(NJ6&XE%O}xk|j#JYqigPZ*}-)dV1{=~gr5@wuN5?Z!vN z=j7z%#^PZ$d!Es7CMsaZ#Ky+NWc5>O1cikMk6(Xt=N_1yRh9knk(~2tsz}9NMeg{a zG@}R#U9Qu4@IgJFup!B7HQSt~-CmiPny9KfgzdWQj^~EJP^g3TM&2~oZ9C3%fs2QQnc)KscQ4p&rm^ufS_QTVTI?!OE*I&ejF<)r?B`r;=I$pfq{YfJBN_=6N$JuP}(#d3aLPqEXcqZ347|i0|A+wONkPg5(2#<5QYI8^f74#gsGn}bn3{rvqoNqOpTEA$l0|(72&RjA zo1puy->pvCpUi()HE0D~UtPUYpAS|lGW8T^oNX#}*uzcwk(`ISU~V2;=`!%j#l`QN ze$Vn2VqKLEq*tr72Nl}SDEf6FBpMN0`AT2k&zZbu@di1$CJ}A+2h;OIZfR}KOVHRZ zgPB9#E-rhxiK@=X%)`~3gm%6`c+7Se`@CN9Qqj_OQIbKp9IXaQJ1f4S`J^;n>=|j;vz=epGcj#e(%NEA z+rpM^j8CgAro_RxH8ZolJ3$#n$gbUJPf(Yw<8(0un^OI9ntsurnV)Ze+Fjx;R5UoW z(&{0hCY8))IW#n6I=#q9jKs z08?2$hMn0Q@CN$U#shbGO;BhUF5ZSn=})K7AtBjooOat|H*_oyOHEF$XEt9s@P$C4$9V@~?a$C_RVJ)$K$cr0 z&>f`2l~$Ce>Ak(tOV?M~jIA;+nhlP#!ic6PkXV`U4_H@644`@O3ay_O+-A$)fq^8q z->gSEm5`myP@_u>j5vRUL(bt#q!bj6dx*s2^`5C>xzoN0W61_xZUT>MgiGeyxTIfj zozc)9=!H9zB{wj&Pg5{4{C{q}Ft53}nL=uiWs-F`_2P@Tc(s*oZcZ_eBaA~qftjA3 ze)$?h=}1UWkeJ`I+F@YJ*Vnhn>2M}44sKx`pOCP#f018o#%1Uj9Q^%zyGcA_2tJ+x z)<8Txs0oBT22U>v(*cxKp!AC1d4l<-m)LQQ$N8z@8$foft*v>uUd-2+W0zpS|DwM6 zI)R0T)_0Hmgz2`pv=kneT&Gf|-r#Zy6$*I9go2L~3pmiQ(7?dp;OpzNqM)EY|FGoz zj%EeQKA8Mp0_-l6}PhK9qbVbbivb$sf=111o+TsS4 zN$!6(QD{0}JYzD9OnU^~Oh6R?^SOnVIe2itvOynREjqqu~U`3Cm&4 zCp3#80NAOjm3JZvj@=OTG@7Qb{|)U(W@Bk=0+=Ln1o6#Gnf1}Y6=2Vo#~U+}hdpkv z5wN)sYHmX{!t6V>V6> zWG)Z(y95d3Wo1Ek*({!`in97=fuE*6zY`wapD~{)Rr~05-*`r(sH7y3kofXDxx_0u z6yy2oUBhInkA_4x&*|w)i0a_5`G)o1$NCn79plqu1T4KSmw{2j49JYu_ugAW^yjc; zhnsVHE6So@zkX>oIoLtCmwS5ye=w1glh+vTvpp$f!@-Fbobmn_nCB;L4W`ay8;`7Y z`V$hd+Bgy~R_Ieo(9#yV*&2Tl6Fc_2JOUsA6`MH2thFw}+BE#^Qp?LTSiFy_(D@#w zW@JRsLJo-bjg1Y@tH^|e1nk2VjHl@N`J<$Jitf=4C@EQgOF!UsB$M@KwRf$vNTtkR z@k3|5vEFlz8;$k+$}XAir05 zco4LSrk{o|DG9>u$i&E~+mG{qDjll|tOWmF4hf0x|GK>M|1BI*_98{({QmrJYePc1 z!~zQ@C8Zre`qYq#DW|3u^&g3B&LRBoo`BE%-;tyKkM*K4DXAJ6Z5637*tKqe>dkJp znDFnqkdUaPr_ebnD=V!Q>+dP3b{6WaSzmYWH_qfmki^re{Y*=11=O-%2!2uFwC?{N z{=H6Ey)PN`m+)ub5eGXr#AQ@?ZLPn*h?W+w0&|tQN~vt=Se9QbsJ(!c{P9CTT)e+4 z1V5d}<>kwldb+ybF5p@bA!(8gegA1-8H;4(6g0H7|4C4WB`^O9|Gqt9Jd^_8+M=MO zj0dmeWDUSJe*gZx(P4jUV*^x5`Nnr2K9m*~_IrRDm+pEFGu6`@@9Q3}aI%KF19^=l|*Wih7@PNN8fFuQ)9{By&L}bVZ_t~w+8oz?Q3`XV_{=q zfj&GnGgDVr2WW){#BOwCKGBWb;q}=2K#$u)Z8z}=rL}f%fKEuDI z4E#A_JctdL=JIrfqNhtV0{r|eXUlc}{0;(Y6d{lENA6g8?0zQDqyg0(mz31+p$Jy! z=ve#X$0IDs6ppn2?dYmp=A#|}#6m+c_os^Tz!qd>oq$;r%v;;`Ea_3;t%d4grabMeF8X=^YMQ~@K71u-u(Gc$vB{pucp)+d2@aMpbcKq%HpD?pkUT$FIXsirOj&_-lC_cC#iPh^L>54 z&iY5$a)7U}p^=AfPp`N{((An}*~kB?$ockw)z${A(*8_YbaXU9{-^QraV$cX z1UAd0)KrMVGnx;pmjnY_QmLt_0GTC4MZJ6f{yhT&1DNeVm{F=oiX81t6@gKl&vm2D zW1ARSd}q}j6_t3`e&M2LTUeQu@-r(X##hxs(Dt)|q^n4rrxun%^fjy5N2GR)N2q__ zs$wYy{s#^&E+*Lm=~S+%8gmtBG1@;d&Vz=Amd^L2sHiA8At46}Wiy`+{p%UJLojj1 zHvpSKdMg}(w1MyMgN!o&&G{q;b93wOY>KnpvR(nw<6TwTMzzWE=xj!*cS?0kPJQst zl(g{1PeVy%D;_U#pUXzZ8x_{VV6dqo<>@k=^l2Jo+55_>N;0zYs=xR8Q!W9c-sjJs zA0x-ou&~(oDv%E^{f`TXd?%@fV?d*1EB~^Kz}3|iq+A#gdw5uw@qjsyly^SY^tiWEg=mL$+&P(+{`5-zyAsV z*V@J5QbjM@ks^^XE%k+{I?Y5Wn0_FsC_hgImOxVr6B`)dg{Htgwuxg5LTJqI2(CB1ffIaCUSJTtr zLVjuNZ{MyOe*J1MSq;j*F0v441fb3VTms>DjDb4tjE?orIs>t)v{WBR2oPM{$)fn| z+S>Ee?Gc?8cQK5zB8{2xaq+l`HZe&_%$E7Uqa*8;)+u(A(MAVzuuFt87}e$F8I)N| z-yMT$E$656)lQdNnm`C49CvcXB_!}Y$hV}DKK{GUNU_8=06+$Zgj7B10n!9mb&JVZ z78(x4ihxyY1nTkp~)5(kd`dK)9W@wl=|# zpI)G(>B>!rBdnYN(dpY=Q$zrUV|KORouEGaGh8~^*g1?1}1 zhI{7fcZ#48iE8tiw3og>=^ReuWd?Hx8*7Hm4i0>a+VbPQ_^K)}rZNrP$S8Up3zdlE+5z3ERl|hTEA9m2r z<-q2vwdr=|1mTE=y=gE#HR9wn8;&=odER1l-6AF}tssB7EAU3Fz_iBvK@9ax%x4b7 zj?2OP6qqLnCfPqSu^Bb-VPRv-NQqhQFUA>x>^7S$mun9}Vq48X3iPl5V8X-09ppS7 zUhM1Pj<@HjP2~iS4GBilcxx&uSQ#0q5uv!m1qB6TNQNX4Bv<5QWU@3JsY&taqWlk8 zS?@vCI_yu=uf^`~?^l%tti3VQH8LvA&KBxaR;Irtxe4%fHydr2jq_RC;E3`6{rk5K ziF8LmX&NzBLSnKUg&+eIRBrpaADNkD5f|1iF98yxVPN2YN6pH~>hsbL`bKv1~Z)rBKB#@ykOO{Vv1%J5bp!Rg2Hw)7jjor!oO(D_;;BUk@)g=`O}a$CV> z{X$;Jz0W9A4=A9*?d_e6!wd;7%F9!NVG<mal|%NxQs& zO-Tp{Fj8=F(bC#Y_2Vth%}S(iY;W@)Utqi9;F6G}cXGZbWoM5_6{uIHhhI_j_4Qr3 zxCqPezgwg|5&!aK^5u)nuwUGcH4|JoAeYNXP0mZsDkLlxuAY|2uhhZNq!%;S z-!~W@8vf7|K0`Hkp%fpNSx83tPW#}zsB+#8Xjr<2uWV$yxM^t2rY+bR<;FnmayW#) zeMfyYAPW3jGlgd>LBx)Zj-W#&ayig=_;7P`M>w4A=Tc>o$CXkHZEQfxwA+e(Jn##B zCsMe$8X?^C^-WC?B>bND%`VY-l8G!XX1fe0*B1x}Ru5uSPt2gvF&YZhV^4(O7uhcg zLa|pExFRNIX3MP}`-8g^*;?oyXh~@)+07<*1gn-n)4IEH!bLMRHC5QC(7R%Slf4t@ z5QN0Max=&-FMqC(1$g?P=R?Qc`I^x#9ROv*-*8%3 zSd2GF%y2pE0lF0Ei5t>*xYup*9p~65#(zKA_W=F$cX#_^rA83*?(Od{WC$QbX@X=V z!M7pJ8>P3TS?hQkW(vv7r#CeFhANoFDf}Xd($0!!S`Ja3;nRqtuZk%qa*1B z$6i2G1U7)Qw6t6ofNA|RR=sNU&;emV$K?of2B-x{am$S!H-6Qbp)92IUA>b={$E7l zi_Jf{?X-AXo``hkNvA^BfPIUYr#!vjllmSFp+n4FI;zLht2T+;jjhe4)(0{^Mm1rz z*6A{DADt{`K?NaZ-t3Gjd2x1738_h`SY%XHO|t#NsKhgB;$vbERl;Ht5;OT~%xr9N zSERVaiR`B1*~VqYBk7x)cyHbbKKhI%>d-$hASxn)M@Rr1SlF2dMWvh>FCV6|lzza8g!ItdhR%^cWcJY70y`Qv8~|YRIS$VLV(!4tt`OJr zJ*6aaG71V@F6bu-3G>N(ITo9{YX=&pLiWQL8mdnO0r`(7R5#6kSGhGagGk|^Ep1zbV8T3?D!*ILcpo?CV( z#Q>#5qSL}ku;|Zf2QsE2Wp%({3TG9VA7UyB3kuZK)t?bCZ2*(iB3by$cEf=<%caKH zAC=$(M(RZc1ye<(!YhO4=jXs#wMV#gB!6P&FFy5RqYNWij3y~lmKq^ z>hkhqEF~#3Hw{HpfyG!w5JFs>-wV%W4iA}jJ3?6LQ0bz{?HST1ha;7|{5L%13}BAk zTZ1`>^~faD+xPu(*k{KiW@6#tX`Y1>ql-z>T_&#ahIzk#S;z@mih`D3f1t+rcs02a zHanJE^ziF}k9t9jesQalbu0aN18i2aqo9TZ8A(lDO-c1({4ZAXq@+m&rML%NLcH{Z zl*CwiEe&}ssku)w5{jQy#pQV-aco`_+-f@|2nu?GE*BOORzm&u(-{jdZwqjDGNn{W zq@-N)ftvZt4}(QXQL$L-=A0$X>>H&_ifV&i17OGjO0Q(9@~qteKCa8FD{WxXpkb2N z7SL_~{=-lDEA{=^Q}k_W z{upY^ymd6{0yFz^1XL<{C(iLB3W^^NrSzzE+)xDb7D&RG9I@!t)vuzWqSx=Tb^9kl z54;&3sH8UrEzyX&(!Kl5p_t)1O90@mpE@IpyD zsX$5Pn0w8-#RG|oe|#LJ1@(UJcsEzzOs~);>@}kvkWq5e6)8j_h+6=L2R1sX6ppAh zR4+UmD%v7h!3x7FH-I`Jt_Vd%Mc7i)NOObTs}JftJ(lO*ej&~`GgQiNwnKviTj*um z8EsZS6$X=$ky$J^9`p-GoLMtqDn#@S8vkhv!c+GyFSltrhkVd(p3RlOpXd&|bN8U9 zsdZvE6z05U;7S`}y??u&f&2CH=tY+_fIpTaXsphC1GK=pH_)|*EmNvDnY?sQme ze{Os{*ew%hx_Hj2*@;pnP2S|H($>}j`>mgex%nxdb>=4S&0k;Kp24-i6z}-;X!YG< z6tJ>3OJq~(=_pl{B^_@&)ARs%YHVr4RUc{eH&}PmkBo*<;Y#EA*3YVzX9$lU0Wxdam zN@62ncFd{_7GJsdsfQ>&BxYo&EY#Jvd7f2T-h}J-$i!_RD9$$r;yv6PWq;i>ef+w) zhlP9X>waDT?QSog0l6SCLv&<0TeJI^?KNPNYRr|Mn+6320@%sI%A6;Rw@`trGFSS> z_AGciv;YS>k}~17`W61=0?YgyHt)H!LmllOmRPt~x$TG36aLuCStT#rwe837QsV`k z%YCJm1O9M&>ONb`=$G`KN5B>&Ne#-~!=3actl%N%SYNqxZnXHS45K>N>jp`-bU2+~ zrVex#BToe>X_CJ{g8}wVLPEl32<|00RY1O!v2>3iHQZD5HJ~KNE4B+jnQ)5m|MU5cZC+wtNz&krXAkra8Z*KN3 z@?;VQd_sGE5GyIE4`C}HYPK1)r>*)+*Z##sDPp(+D^_SAMXRe zn=_^`T&TZ+Wg1HgY}b_Rl!k;10EPg}J;3fPrTeEZnsTlHlr!M-%cgJz(hDHKbvylq zh3|A8n98(Ul6YKfn;Z{zLPGxF2?=$+6Y_Vzy^P}S1g^qaWB0r3vx9{?z|;U=a}v8X zpXEFlGC#V?qj!Pd5m(w22_uw~SJ)93X`YSeG8xUNuC4}5W}WGTzo#|TD5v}7N2y%O zR2EZtKxvugw6ySE3Qk}30tr%q|_+sKTG5gbuAx7t;uloA>>P7%^?X>W_ zA_6bOO#R6v;zrH#@TB>#aG#I9G?|L+p}Av5*a_8Lc%|Tr(X&-^x_%p_-F6K4JuDl9 zuOvn*ELC5p;I;jZH02uWvc!JBd}0X1Wm1a60u+w)m;5IOoEa)PBBi zx%o(@mK!z!%g2~l+WKel`stqJ+fT`XlZ5z?tI%!+Zal05{k8bV&u;b@V|zqeMna?9 zWNzKKjC$!ShsI)W{o1`xPk0&!{^k4C+lC3@_*X}JogV3wcH0b|x4nmh1?z3@=S-TM zbj+vFnU+%9!}0Au*EO_zdxVlxanDKkO^(Wedxl=4#_?ua3dF?yrcl4ll*7TY58Io& zI?EZ|<8dXJ4fO=pv<}=CpLf_&wJym`t6`k#?!zOb%>cn*AXcn+y_o=jgJuuVVB;#N zvEd0!f+%D(npZbBRZf7{m3Go;cJ8?4qgj&;TxoK$_LqqnQc@%YW=Jm@W%z7s82a>t z@i4GTf_|^X7vKp$n61*NgM+j$UmA$Uz~>QpMj}EG+t|pZg!WoGg!gb9OnHwv;~z49 zA&TM|fNR`%dGOC5{X;r$lW$H=qyH02;X*BU%==zN^2K$IY8J(ZEmtEXStht}OqPqS zk`=)>5RO2Mp>sjQjiOU6|JoId*Nl^BG{xsUNs% zU)$U~L+mOPem&(wK|#kR=J^^xO`f^)dt#yrXp1qu<)j7Ez{LHzL9!2(VEota0Q1|o z-&^+k1p5B!db1erYrR-6F!z`jMVJ|UT&vCs62{{Me^Y{9TaakRjsHROt1&>YpDJWN zA5M03{E*6OupXM3aYF2j#gGi8-h_w%_uUE!m%YhGZxrlhK5H^WY;9nBBz?|Bkb}yN z%XZ^S8A&Sp7;T=80^ByASYm(U`fMK%s779_{qC{%uV~22qn_7Pe@x-<*TY5k0KIPq z4l)cRfLNJtKP?P%Af6#lM+DtI0X;r0t?w2JBq)0I)$?Qh@&*?Lc#!9McW8NaB@}83 zw_tBBb%dEnD47rvv*{s9AsSwDZI z<3KXrFqt_oDWMVz4G(s|q1Zui$}H=%5VEr6dO6NEi!rN*3t{(UG03ad$CajeP8bb! zc6KJ{P6S_)jOlnmi8zNOm7*g0c*u0g8PdP6)Ty@Zpdl;MJg-pT@HVhbY?Y8f7;H3 zeFvtOR!`pfcL3170O|`(e{lv@p;acfycDE5m zd~syF%2!XAFB4PcNE)A7lj{}GjP{6L%X3lC(Mi%y+^l)HprGV=BJZBc+hda@CK4<- z!gDi9Cd*1!Jz}>GkUvfReuCN)4k2PS!^QKBqrW-yPEZ`pdWo-;RWN(CIe@<`8v8Lw z7|YFGLf1oF932aBun@0)`Fc2NSdlLC=j&K`s(TK=j>geznDB%Fd)asp4rRWGjIhpi zH&mXSHB|0J_!e+GQ%D_*=Vo{z(JaVvzhXs|mv|Q&m|qg2kyiZve(!1cDmKY;MOImBaZ4{f)pibXz9SFptj9 zp)cPlVQaU#i}{UZ$<_J#KCVF>|DK-CaOf;EA_2TK*UoPEjqd)62lI+W1!`3kc$oL?r!)Z(lj?&OrUPMmyvIM~q^uMrXc&_5u*E zK&(<>rTksuIM%h^T(v1qE^=vUDGPGgHZQI2U;<-IDWH_j+J7Jeq-JoVm*GMsPK9`1 zK%<=}4N-qu*X9o}ZecLd>(`@yH5gmSQ9Zwl?p#qtS!sOqD3!~BklT@tJ`y7K;t4PH z9gBQQhqiQbE9D%q5>R30t4%v?#!2}3`5is`sb`Hdd%rhD3ZBE}I*o9EjwU53nbT7W z`rdN0OA4`9(m0o*f`a~bPHbQ4lwb=3771TUTn3=VMzTsur1U8y6F%)eL79_?qtkU0 z*457ti>3hVT6sj4V%OwkOf6x_o1}nc3huO*R8V(i)7>#;tLb8b5?)sk$Fgh^PxVto zPCAE)N+#PIbvc%x;J<)Yj5t1KWM|KN98h30r{h^ho3*$Itl3%`HC9en^N^blOYR=c zbH`xP7$0Y9wi6NYEz0*?vD}-KbGxWj43IH0+Kw@*9dCv`OSLF)dAM)k7B1Tsc95uD zyMy`BIyjsaYt)1Xg(q=(jDHspIw#?h#xn}mM>mR&Ey;!s-?XPNaK3!GQf9+IMy9Eu z&c#C)W0n(hCVN-wVLnjtp;JhRAK$gHy&Z}7?wa2%m4IMZF+Goq=akm4-c~h6W^UH8 z)phem>7Z1*)x>y(m2&&0%3{7gPMruF!YdVf5VWodIEo#wgCx8OqsFDyLw1Xt5O424 zE!=hK{BDg7H)#XfPK}%hJ<_zTS8VI^nUFs^yoZkEw3*2RZ{7lR~pF9&WcFq;nzQ3 z6hgg7yZ^>Bj5iYE^91{}*-=AEtJ88zaXDag<`6~@y~~@{RKLb?@l z1l2H=HVPi~jgFSb#FylM@p5?F6Ga-}cCow8=4SM8k6iHdW}`0}*hfYW4K>}}Nhm34 z3WI|KWjm6#aw;KsO}qHve=xzhiCW2K{hntuM{-u5H;rgSPCq0PUV|}^D*R_u95l=3 zQ~N1t0JdBr-^N59fRKYJjNlBN$MMfxE{C=YOS$zP-M3!u1e3Ppr1X7ApIox&5?cX_ zz$?+^RJhmF)CAhHGawQyvbj*8i*I9FwRbRDX8mM0ZvpoT6M9-RUZq2=*g?!`ONKQ& zH#deqHkK_4P){OIc^!+yj!`eZqWjzZT$(CE3Gl;rd$S7Q{_NB5-mlN?r5Ys?8I&$} zHDWxo@;!%fHDB(eR#q~d+Tp$_54!pO2s!Ct$0dMf3MTVns(o%iuEpyiEtLh#kLS49 z=&N&A3_7iY(ll!cor8miSZUSD1(lP5J`kY}{t}V%3`{{6RCXAZuRJy!J&(t`#e!z) zHcyInhc`g)FL0pIb*{RQ`co$BpiS$DOBv62tB2clO9uWCOs%6U69a-!Q7)>E0k=&` zLnB_ds4oUoJxTrRi^CD+2a^!VL?%DpUH@*)X6IvI-v&0aywvC|U7G1SYaK81`Eb0Ku}~)$2HZ zMyp&xdyP&+=*iz(GBDXp`oUpXseZ80y=exdNpnV`JeqeSt%(YmAZ^0Xh!~P2n6`D@}t9AmLsHV zW;-1XHa9}02t|ZNMzgGzk4~ChC8V{%*+UptRh|i6)9`Ks@zoWJnMFUb;xF7f^QbxI zJZEjh^A~ujoNj57)Y8*b3rZ>*aw{L4Q@B0ng~ZhS?o+A9>g_Y*vvH)d7K1%}a42Q0 z4GpC;G3%_yR2Y!Kz+p6`*3R1~D<`+RytE)tro(DEtr=#QB|FA@G;%w4^Vj#b&tdF} z94~;O*X`IYn9q*a#>}aMpyoU00 zSg)zGlupm~>#{-S3XyOdYice6CfS7Yt3C#FRdNQmpD&*i4m^5_WSeY@75QJqxv^sP zsId}xOn;TX^J#M29j_9_rim)kYKlu5{`E^RnUyu2@y< z^3=Pzd&2_4X15Eg>uYF20|FzH&C9B>!^U-gHN5N$nO7pSK(UP5<%Dd?bl3R%-uB`l z?|H%}Z7d?T4dT`IumvzHNUS(x=EYOPdj)lZguj{3G*j}~gDDV_VJ+?Wmhb*%Z3mZ8 zA-vsYSYEfM92wy-l=fP%BY+#*v8Cc0a_8=+nIWol3P*=l z?&A3)IE^6(K2;*O(}z3X3GaG*ns zzUbE*Un?3a3MnYdt+h72+>29uJ1{u-HGr9!>30{oxE)#*A#48$lQFItY~IaeGbv1g zG6KW>xOsFE9LtowafybabclX-PUyK}j^P-6*m>cD<5kHFm@AL1l!EB)B-Y~NQC`9S zaRCo|Qxb)KG&EFW-R4WsmI5`L!lv*y2jXXTR$`q1wfff-Op?N zE@u;W^1lRpNUgP)|3tzLNEnq;ZLjs~8h9$V_BkELUx|+Efbuy%SZb-LxL;o4&My}| z2$)k-V`5ry!n49^n{|*T3i%Zi)tPEMlJN6reW4)Vkaqg|8Wqm(d^=cxb8`XL^USqS z(Rx>1YNE1zUYe&aN>6`qqg0V=He-ZJ)fK5C8yXR^&KLZipHv^ms!<>%05eb3ptCX7 z2bHS@^EP^7)eEY*!iN--;TvpB?S-IT{pwlGDHrmza{~4|ZByKmteWb4FKZD1;{pSV zOh!7oP|L*ZN|H{~SARU!^b4LhG+b@Ph^65Dd>>1xSRi2*tZN&7!iEM6mL}M8^)$l)cYOEOz#~{Cui@ z)>$jX{4p7y2_Ju8KNo%Xx313L3CI=_AD;ja#k#*9!xhEE#O5P$lVpEwtbNl4$ALxE zlU@n>t*`9Gp&{LqE7DeNXLQ?LcRODrsHaS|G&OhjvtqPb@%G4ncp`}>B$k*?)M}8MqI|1DTA{($av{x<6V&_E;H5 z*Tsfsz`paQ`*{cX$|T&mZPfnG9?uAJnT&E|1nEcX8EoGcz;E z2M0^#%~Ufs?R;{6#;(!OllS}VTnxnd2)>`+N1UCR#%E7e+N=rEn2oAu4GDyiVdl9D zH3h`SfSBLyX}5wk-3} zZi$JG&l9<+Fhuof^w58A)rkSSc(ux(<%4e5u-sL%hp+<9ii*qRN>A zy2VV*z?!bE$MyZpRCvAbV?+708LMwTXs^07sOw9!X?% z*pCadP=R!1jn@hwch(+oHG4h8U2Sqvrz`bjIgnFO5JT{~h#dg2T#%2rQ~hX_C2(aV zua`wP!XA;5;y$JyY6k`AMEvUfQHF&Fpye*I!X~GMT~KQrVm;9m zHFXz?ptgX(5MO{w5n1)@En?=%c8c{ydqQUg<@iv!2kAd56qL=9ks|`1|WZp&BSwS(o<3-4UYloS4HVtB^C>g4g*4> zz%uer5jd3UKf&_*sQ3>s%h_?~Vgrq70hqjiZ$~PP$7N-i*LwMJ*lRi2dFU(tQ>YXH zQvPxho24sIFKdkh{+_MI<4=D0*+$Out8eF5-sJ`YiiQbAMLBH{N(5A4@ShsM0cyVw zPk$B7uEdU4yT$iqQBfPA?j*+z!3c&;A(BB*$wa}GMnI_;V`FCy2Fgc_U;NTSyjO#@ zCot*3LwL&Lb6z6ENeY|}5&Ht$mqc9=P4^xU33 zwL{n>i|eTq-KAA!7s$12ZXN5I6GpH8AKKnBuIep%_XSZ19h|=AS zba#g+umu4TknZm84(aahZt3O@w(kGE=YGzc^WykwfA)u3Yt7%BV?N_~z9UJY+H6%- zOkcJ5=#%5CS2@rHdGPSTxsx+t#HB6P?ckH8vQ9Gsrs(l9zV;zwh?O-~kqQ_Xyt;cH zHO`iruCP(hgX%g`U>U*J?e58v9xzqmc(xhw7a_NknOeaTo^&OgAWlPHTMgNaW}T2c7B zi65~h-8m;%jzX8>u4jFOH(BIVNtT;r*WvVFgs($z z1jW^7hpg)Q5+waRR#UjHdkfntjXQdmO|ecn$+l>d)mZ< z<*mc6`*gC>S)eAmE{@L&|C#*G#wgQiSv*p_`MkSH`6|8R$G6~v`4*w5koFjCspe3C zNINCwHZ(VT6~7oadMBa*?J6xp>J5#-Lb}yFI;G-7b?v`X0_D-AEcSMNztY4-I3T4e zhlPdhAJii}U7n9T_`17gr8VhSqDN1uJ-1$Y2{#% zj!eH!w~pa*{6U>sn3`W>EF5sp>!?02Na1L6{HP<0X=3TtmQ1sYZI*@HD(u@U_*EnJ zrCtp^+cy>&_MfaCh_iP|6YP)kI35};HDlN0<(aFi&txhTq%!l0yf85_u^R5_`8LbE z-9zV_79^B<&-_JzT9@VSaRADh4GE8f-PM#zeSJ}KclbJ^g&rT@q{!B~!~VHTZ01qJ znFm&SNN{ijvvu;G8#lA=Tb-tz0D58SprwN#3s$l2yP{R)3M_^%t#?b)`gJ2L9E?A3 zxhx3vb`)hqCwLY`i* zXEfLArFfQg=V|8_(NJN&!Gm3DJHq8}pBuam1>Sx>vR$i-cGno{iCv*h{N?L=peq`q zEFj|Cwy#j*z^7eiVb|K)zCJLR6q+DjK$OKyBgnxy(!Wl;Hn9!jRY{Ioim+8uMQ5k4 zjI<0j@_Rwyjj6JSzN-e)8Vh=IV=E|8UfiySACA8vGbM`_7OmIQ8!TnolkuP`u^2L2 zjwRZVRNEdYZE*6~Ec`{66%a7>@JfKs3wQsawAC=DlH}0Zv~4%F>ShZ? zYh@71@882ITpwu46Id?ZL`ar5`;{2#X4|6ADA(uj zhi-1Vg_`17ao$fJA*Q*V`j@;Bq}N@G^>&q9~BQ-EqV-+ZhmfS zbAMeQHE#O>DfTpu0kiG<>e)vZqLm-N_2`IPXYP> z543nrb{8v`-{CVFjpTxy)q~%4tz@1AxOh?RU3LUSMTvYtZQytr5f$d5PW38QIO^y$ zS*8<|zEu^emvyY)B%Q=l8@gUo+5QL#NwJnGF;NjbDkE{t&j^C#S%E@zM#lO-_sOXu zR{D~J`eVr64?`mynr+~v>SKHP6M6rhB3uSm_xX5v0ylz9N;CGM`8iN2thNho5BiJQ z5L_VNoLLDyMmWl;snx9Ds@@*Ix}CL$MZ)BNywdP`fYZ^UztOM5cw z!}=>|FpLarbbY!){4FA)qScVR=@BiIZnJ4kRGizO#G17n%`R&dl_OsjeQDo}Gg%~? zGhP^e!nJtcEYtRCdv3b$6{Sent1FMCfPi3?VhiKAfDO?Wv(tadoHki)D+BW0PrIPl zFF&)v{;)Y^8%oV`lNd<`IrzD22E#jpnz(%0hCA$~V*jRJO7@!lVsNZsS(2wGYn zNiuLSOzownO$L|bxF{Y3S&0XPaFInHqS8c32~h|8r`3wg51WqgdiX%)&8=onEYnmA8syFE4JA^2O_Hi0bI@BW;4->5q%NF7|W?Y zAK<++Zh`bb5j1LJ8e(JrnxYLfXQ=Nhv(!C&<{w3jp^K2T`dL+LEK_@^{(4njShxcW zZ5WuC_7wr~Z$3RCGWe*`e=ayd@Y&<$)5bS_Oc{+&C;o2aUp)xY>7qFR+mu>PWG#NT z_1c=Zj}WxwRmxXhnAYVuO4IQLm#(G1Za)0oFdyMV2r)p@LXZ66Pqa$%WlOW|t}vz) z=0H1%Gv&C3BRz~Gw~6@F!k6KnY!{w^w}GZ6FT^}8pC(%^zHD!);;XqziOU6?Oe3TU zX+4$T0`2J$H*i+T)4auh-MNipPY_R`7(l9Uxp1JN$VYRLIuC%NlZWOOrS9pefq?-B zHmSE2N-SrFO2r`+^y0dm4dUKXSFuhfVOv&OvG6n{nB#el};b&Kqz7HQ|U)E@R|$0eG1pfGhv)Q@-4(w^pDZ%{yh~NFoFAo{r7{ zSI;WHJDn5|FH22*M{5jrIKG*-9Y=Dd3Us(gP>E&z-R?f(*msa|EY71JXMk}R;NMF@>;7@yV#0GL0ViUM{8Iqm*OTQoTtx-BCH#hfw zgBSO&Bh%jca(JsDL*{%=DshXPks=%Wc=wEx|$}Xd?0b!$Skvy1F`< z+<8OpgN4?IP){l-jDmX~fHGucWZoR1>0p|C%KaiZ?eE=u$=&>-uxYt6@X4w#p63}< zui-2hw70%{fwUuV&rJG0Y^Kun`i^sRb4tY~4ECfnPP7&~ejtezLy-s?!Xyn1b+=k+ z>gOqBWF)uXz`z0>kyo}~y|73$83Y*BYmpL(wEuAlW;L0x1NY*qS9P$c1h>_e}m7WsU{c5I~v7g({$&8ou6foYYzfX*`h$&zu?Y+Nm1uU|yNw^11 zRN717so4&SnV6_rDNDZPKMo3_I_-*nWu9@LE<@pY0NQ10wa1S>8bxH*c7;n~?H@K`K&gh^E##t;uGa^b$_^w={|4@MI&sx04oHq;G;MHdgRc{EY~q~VUAiE>B~s%t*dV!0|21>Wqxs+g22xN&$G#Qnzh4(iJcEP#=R5vyVmSYM+5hJ> z&>2S?)~F5tzT1=6P8Ggt?e|>igXB*lqoYI(eG3aVoy? zpLAHWJ$3OO!`~kdQta++M)9Fjj)PepG&a+zm4Q4y zU@$p`SfC3slb#-1uZ&2%SKy%FJcYzCCJ}C=y2bM0W@hFyhyyfFIiu6$gMsM@Cn0FmVnH`VSo4-^l5@+&ZYWW*o3m;;)wv> zBZ8vl9ZI=_Lk!<_XZhm%#Il=K?$*tl27~F=D>IjH$dYT#FD!iEQByk&2?^XDNqbmP z2sH_Grv^+fwIz|!RIEx@pG2E87vB4i4aZA z;}Py|q+E`4iiO7$Wd)YYrM=SL*bW?=oE!&}r5aqfi2HRr5(Vt-RT9QojmN?3jjuS< z;Dt>!DK01Hy0Qs8yEEHu10kXQWAk}R@ULddR|`u>{MsM2+~|N0ZlgAsHiS|%+(=(L zDK+)Y`!>ii=`W-Y8hT=o;E~`C#yMBO0;&R#Pfp{q3nrZ0}}DWo}7rkXlUxOQirJ73**3l2gytY1%(a!r?51J z=#-{vY`5FAyFD?9Bx1!YY>brbF~Gvb4;Znt?vRA693P%1@tV4Jz9BUnLn$zee1|Rvq|D+!WtBLsf_(ZT+zIAeX zDRlv!7a70{UH#HJ?4z03lX{t>yr^>z1&xCTDw54{Hs-0mw(GD8VXPGdaY9hx+7asR z!uCK*Qxmj8*J47Y5`#`Nto%?!v#+zds_K+=j$u$Rjujyxp`aEqiQ~~ak6WK_hp<(^ zpFPnj`8$f=$jAshblX!!1%|qgggWU2=H|alCP}EY?e-|%Nl6uydGO~I0FJl7BZ#gDr>!F_CM_(Nx4w$}}#t7?y ztc=N=oPKbM0re*klXz4gGUygxhpA8J`Gt-~$hugR6M6G*-=3zsk}V>0=1 zi;D7~o?aS}LSG;9A+fQ;+}u+*Zy`rlV9=>(^S#VI5SLa?=-s=%T&Q-Ut1r9p7@Mp& zqMgq+Hxa`)_ikfe2)A#H6)Dk)dJz&$vQg7eS2>^Qz*=HuEl%l<*8THd91H#%gJ*oh z&u_D7LU{cA_(oy7YM4H zQDB(N6cmM@BBt5lun%BIWvn zmVgvZo*=vXClNM?qCpqa^?2pQtGexw^v?EnqMn>>$j^d+$d#ikBO#$pBYUfaZg_ZD z?30VmUco z8uYOr$ZbU2+W&r+xG11{u|%&3Z>@S-xBuV~VtdqeLMY%L6>DFroBt=f>C|FBg9OKG zYaj()6tm$HM)(`1=SbpjX{csq4oL-DZ!O}(H_F2sR7rrw_2YNIe1;Dl9X&dPUPo7V z&#$Ai^MKt|R8=(^*)3n{K5}#Oj5;F1FE1}oQnF}rsUzupE(ANy9&7=DLEq!Qej+`C z=D_b`&@Bf7GzrdLeh{L98--mnpy%3JTk{GF;|U%iQIX@`HzVe9_?w>I4^>i1YN}5cKmoO zPo@2%x%PH<^-#B3z|ag&NIjGSw1ycRCIw|^&VZsR^&>Dg)=o=E8ZGjHn%Hpk7ZeEB zhc@}cc_6r+9a)x})qNr%JD&UbC`1CUGr~YW-FACukyEVrdr%N_qrJGZvx>vOvAGu6 zL`_b5W=m9fbTqfEpHhVEP$6$fFL_#AIPwgwyYLASqg85Y>2Z54?`ey^p%c6~XNr698Uw&AVsSg|oB71g`hMREjfi>y~U`K_XV`_d{?j0&E8OWQSUtWTmC|Cq|k0Jh2i2bh{%ZBqa@oDmiWuPjDP< zOa*?s=i0em{ym>@5^g>uFPt7}V*q+hEE{wu}$n z#CtfsS7Jz5Sy+rGdK#fx`C)c7#l@pGO~hs@Mfa5gbO(ro)BB&Fk8n6UJ7+uEZz9Zc zEtl!7N)-BaFh#!lk#aF=Hoil36O*2hkw&)%B=jVh^h}D!BUhPp`Zn8SGC1Z@{?I2b zk9^_WYOJ;l0k?BLd3Q92NZp&bDK-dB>krn4i?ID;-7eeV-*$hIoM&{djrlum1tE%-u0H&vmEH zwFLY_SsBA#I2*wBR*|YJ7UM@UuL%#Ms(=R`kSTup`G4{I5ga`4EZK%AZJ=&#g@TAi z5Pf18tc_nShaWVzD0*30aIc}gIt2B$?rvuNe* z+bd%~r{V;qX0&S&Ut)-4cRBB=L)qO$$KVZq^%VC|?lTU?K()*jiq%owr!%VxjReHcC(OKQXh#Om4%UW&dVuZptA3>MGl}KVmhsju2x&u^ zs!x@LLU-|#CuS^FvZT+KvJ0i8FYTcxx@p9@x~7LoIC3Q+Jc{ zObkrw-q--)x^fY#2rel}NpENGuBeL8urQdDFk5$V2E!@762FKn@$m&x#3aPq%&nBG zjaK1usDB6K?`Kwn?KMCVDw3p*sF?Gcx%U7ke!PQ>((OW0RtN7$!U>8B-4=CS$`qGo z(MXV!x>f?n9S$rKe_8DA!dXyh2}J9m3^{+2p`HP|leMQPuDU?uhio95$z-y+<3{KU zfaDP^ybPVSS}FuYmK4+8Agx?!`B!e|L*k>yU}L0Dcw4NeQFT@O>VEGHAAGvRXXkPL z{;{#*Mn=!1CW^`VyfC$p(!YNG-1x}5sYZgl_sTbA`oZa6EwlH;XDmk=GH{Ky))Ewr z4YeLVY%0NodN8CpLz$Gp`Yo#6-7WOJ;3b;Az!&NKU z;pLnSUN>$;yLNWRJU4&hECT#EUrBizr#M4Iz(gc5w2y6opZ$cDVhPL=8?zsF&{p~_ z&KrM0RwgxlD9&{_PYaM~KT(N|;Z&WDBggEBs)>zOojg9agLnT>;yR1!?X8$+z+$Te zj3PUG703HK?%$0LF7fAsE1d@SfsdL6BNR7Yip=s&oCeL zRF>V6>ICvYbFKi=5S0#%$oGIWm4aj>snOI9X49?6_kk%LHW3Yosk9Pu=h+;E3O{5j zteTD44u6IuI;E1Mt^SmrMAocKwQ8p_y5(2VCnJtJMOAPYwu7nll6O1p;M+>0B zn5b~{Pq!Grnaq<)xlb<8dt7EY0gN>I!dG$f!KG!i^fR%wl5%3(4QEGC<}CXWGv}Hm zy3+nGLx!KDn1XfGCI+Bio`{0tLgu4%(bZLboS`R^$3!@R+KDNa zmM^TWeS^rjc-r5&@=W?jVmy7C6dL--r1m}$k)eS>6CD2QBj!Qo&y?9~fFJz@>tIjxbk64 zwyFRNQlGi&zqh<;jIf_9JwQ_m^M0yQF>=PWyX+?+Bawt^V{<&v8VG*3nXI{8(M7}a zgWgNZ$x3rRDJdUn*Yo@=2+oWTr`e{6vfjm`@94fd-TzXPs0ie~MA^OraGrKqJi#Qv9o750Ndi>X& zJ3t~{O}8G=eFGizGE$%D*|RcUKH!KRKOrxHcNf~X&nBD zy~Tdk{Y2W;WT?^}j_3gEv6=^aZcMS(8pt|KUIsu&;9alHb(UJYF`42;-vtj)yUYKN;74016=kjqcCS>QX zMIqJX*jV|E0av6NqjgtLk!CD-gn?9<_4J9wZsRXNWg1ycH%i~G9vOnIT(7Tib?Rao zu{-aZuGbM!SPx!t+d8cR6mp)ke|%GP>yDInHVdF9sZ+5A+(;iCJQ2F zZxW=4MSXDOc|5QWd%h@&Qi%~O8pA_Iq-Y^L%7P?8$Fu<#Hl!AoqJYGS@&oQmNvf(9HI zOUKZOtgE?kAG?5sBvNwtB=H113#TQ5MY)U!}hq&*=Glv33753 z1yP$&=@Gc?3LtWml7cfc!HJVnM!w3n*s&@F2Xg*Z4O4#1X8gcscDtV1w-im7YwJhb z%T}}QXbr2gJ~x!dMM*$U9s`;u+}y53pE&`~o2aN`<6}L1&m!2yL8@G=5v?L|J-Gpm zWaZ|s#l`uF+I`=LbClX^bHp=(zvo-LYoR>6j@9W+vNE%#r^wl^s4s(UB7c6>@hj_- z;P-l%!-GSSY_?tBf8GVs=J5-5`B$QzmtrxT`~_hVtcqot9GE@Hah+BRJCAJ)K76>j zEj^T{B`{uS@?I&d_#k{R1ESd@zkp+s=mq?0@9Mwx)#CA)x~zrm>>eR-mC79YhKEzO zKlUcP^&>;Dg5hX6dvYMPqZS*GLW?#ElDD0mh|66~UnV@beHO5AX~`hNmr+tsMdw_8 z#(7X?y-tCGdpG0@9n;z2R$LVaR6`OH<{PEI+JDe(_gCQK>OnvWp+57M4XNvuKz9rm z8Jo+EBMJDMpgWgeQgG*TCl0_|ffQmoGuzLg^gk=k+HYW>`kUFQ@tya#kzdB+MMQHO z=U_t?3@A35Y&yH=%@1eQ3ztY*+$XEq!v_&!(cF2f10UPa($lSNHb)z9N;YHL0B5*1 zRz!q5K&2b>?VEb6cWYN0$W(oBbU*dMm*8V_*2N3|C`ohL$YN^pLZ@S-wiaKh*eKbb z=_oRR?bWM53Xw;wM(kWbsrl5OBKm@nacH1Vxy*otKTw)4LEHj#L5$_KcU_-QL2(0x z_`$s$r@C`{clXyTM2~3R9&KSQcSV!1jCoW!o(<7?QyOh
_`, for example `IGGY_MCP_IGGY_USERNAME` or `IGGY_MCP_HTTP_ADDRESS`. Nested settings use the same underscore convention, such as `IGGY_MCP_IGGY_TLS_ENABLED`. Set `IGGY_MCP_ENV_PATH` to load a particular dotenv file; otherwise `.env` is searched for in the current directory and its parents. Existing environment variables take precedence over dotenv values. diff --git a/content/docs/introduction/about.mdx b/content/docs/introduction/about.mdx index f2d6b620f6..f06ddfe9c6 100644 --- a/content/docs/introduction/about.mdx +++ b/content/docs/introduction/about.mdx @@ -50,7 +50,7 @@ Historical startup screenshot from server 0.5.0. See [configuration](/docs/serve - **[Model Context Protocol](/docs/ai/mcp)** - provide context to LLM with **MCP server** exposing 40+ tools for streaming management - Optional server-side as well as client-side **data encryption** using AES-256-GCM - Optional metadata support in the form of **message headers** -- Support for **OpenTelemetry** logs & traces + Prometheus metrics +- Prometheus metrics and **OpenTelemetry** logs & traces in the [connector runtime](/docs/connectors/observability). Server export has [known limitations](/docs/server/configuration#telemetry). - Built-in **[CLI](/docs/cli/start)** to manage the streaming server installable via `cargo install iggy-cli --version 0.14.0-edge.7 --locked` - Built-in **[Web UI](/docs/web_ui/start)** dashboard (Svelte) that can be embedded directly in the server binary or run as a standalone container - Built-in **benchmarking app** to test the performance diff --git a/content/docs/server/configuration.mdx b/content/docs/server/configuration.mdx index fb2340bd3b..c416061a05 100644 --- a/content/docs/server/configuration.mdx +++ b/content/docs/server/configuration.mdx @@ -331,6 +331,8 @@ The Rust, Go, Python, Node, async Java, and C# TCP clients ping automatically ev ### `[telemetry]` +Server OTLP export is unavailable in this revision: `grpc` prevents startup, and `http` does not deliver logs or traces. Leave `enabled = false`. + | Key | Default | Description | |-----|---------|-------------| | `enabled` | `false` | Enable OpenTelemetry export. | From 3f4c1b022d574f7319c738e3855d777fdb4563c8 Mon Sep 17 00:00:00 2001 From: hubcio Date: Sat, 12 Sep 2026 02:18:41 +0200 Subject: [PATCH 10/13] fix(docs): align connector behavior and examples Correct connector setup, payload formats, checkpoints, retries and failure semantics against the audited source. Include executable examples and state the remaining delivery and configuration limits. --- content/docs/connectors/introduction.mdx | 57 ++-- content/docs/connectors/observability.mdx | 32 ++- content/docs/connectors/runtime.mdx | 86 ++++-- content/docs/connectors/sdk.mdx | 268 ++++++++++++------ content/docs/connectors/sinks/clickhouse.mdx | 44 ++- content/docs/connectors/sinks/delta.mdx | 35 ++- content/docs/connectors/sinks/doris.mdx | 41 ++- .../docs/connectors/sinks/elasticsearch.mdx | 34 ++- content/docs/connectors/sinks/http.mdx | 78 ++++- content/docs/connectors/sinks/iceberg.mdx | 89 +++++- content/docs/connectors/sinks/influxdb.mdx | 54 +++- content/docs/connectors/sinks/meilisearch.mdx | 42 ++- content/docs/connectors/sinks/mongodb.mdx | 50 +++- content/docs/connectors/sinks/postgres.mdx | 70 ++++- content/docs/connectors/sinks/quickwit.mdx | 92 +++++- content/docs/connectors/sinks/s3.mdx | 83 +++++- content/docs/connectors/sinks/sink.mdx | 117 +++++--- content/docs/connectors/sinks/stdout.mdx | 12 +- content/docs/connectors/sinks/surrealdb.mdx | 68 ++++- .../docs/connectors/sources/elasticsearch.mdx | 65 ++++- content/docs/connectors/sources/influxdb.mdx | 77 +++-- content/docs/connectors/sources/postgres.mdx | 103 +++++-- content/docs/connectors/sources/random.mdx | 18 +- content/docs/connectors/sources/source.mdx | 173 +++++++---- content/docs/connectors/transforms.mdx | 55 ++-- src/components/architecture-diagrams.tsx | 20 +- 26 files changed, 1423 insertions(+), 440 deletions(-) diff --git a/content/docs/connectors/introduction.mdx b/content/docs/connectors/introduction.mdx index 52acead375..77f1150c0b 100644 --- a/content/docs/connectors/introduction.mdx +++ b/content/docs/connectors/introduction.mdx @@ -5,11 +5,11 @@ description: "The connector runtime: dynamically loaded source and sink plugins, The highly performant and modular runtime for statically typed, yet dynamically loaded connectors. Ingest the data from the external sources and push it further to the Iggy streams, or fetch the data from the Iggy streams and push it further to the external sources. Create your own Rust plugins by simply implementing either the `Source` or `Sink` trait and build custom pipelines for the data processing. -The [docker image](https://hub.docker.com/r/apache/iggy-connect) is available, and can be fetched via `docker pull apache/iggy-connect`. +The [docker image](https://hub.docker.com/r/apache/iggy-connect) is available, and can be fetched via `docker pull apache/iggy-connect:edge`. ## Architecture -Connectors are dynamically loaded shared libraries (`.so` on Linux, `.dylib` on macOS, `.dll` on Windows) loaded at runtime via `dlopen2`. Data crossing the FFI boundary between the runtime and plugins is serialized using **postcard**, a compact binary format. Each plugin receives its own Tokio runtime inside the SDK, ensuring isolation. +Connectors are dynamically loaded shared libraries (`.so` on Linux, `.dylib` on macOS, `.dll` on Windows) loaded at runtime via `dlopen2`. Message batches and metadata crossing the FFI boundary between the runtime and plugins are serialized using **postcard**, a compact binary format. Plugin configuration is passed as JSON. The SDK initializes one Tokio runtime per loaded plugin library, shared by its connector instances. All plugins run in the same process. ### Data Flow @@ -20,15 +20,15 @@ The connector runtime operates in two directions - **source** (ingest) and **sin **Key details:** - **Transforms** run inside the runtime process, not inside the plugins. They are applied after decoding for sources (before sending to Iggy) and after consuming from Iggy for sinks (before forwarding to the external system). -- **State persistence** is file-based (using MessagePack serialization) and applies to source connectors only, tracking the last polled position. -- **Consumer groups** handle offset tracking for sink connectors automatically, so sinks do not need to manage their own offsets. +- **State persistence** applies to source connectors and stores the optional checkpoint bytes supplied by the plugin, using a local file or an HTTP backend. The SDK provides MessagePack helpers. The runtime acknowledges a source batch after forwarding it to Iggy and saving its optional checkpoint. See [state storage](/docs/connectors/runtime#state-storage). +- **Consumer groups** track sink offsets automatically. Offsets are committed when messages are polled, before processing by the external sink completes. ### Available Connectors | Type | Connectors | |--------|----------------------------------------------------------------| | Source | Elasticsearch, InfluxDB, PostgreSQL, Random | -| Sink | ClickHouse, Delta Lake, Apache Doris, Elasticsearch, HTTP, Apache Iceberg, InfluxDB, Meilisearch, MongoDB, PostgreSQL, Quickwit, S3, Stdout, SurrealDB | +| Sink | ClickHouse, Delta Lake, Apache Doris, Elasticsearch, HTTP, Apache Iceberg, InfluxDB, Meilisearch, MongoDB, PostgreSQL, Quickwit, RabbitMQ, Redshift, S3, Stdout, SurrealDB | ### Transforms @@ -63,24 +63,45 @@ Messages can be decoded and encoded using the following formats: **JSON**, **Raw ## Quick Start -1. Build the project in release mode (or debug, and update the connectors paths in the config accordingly), and make sure that the plugins specified in `core/connectors/runtime/example_config/connectors/` directory under `path` are available. The configuration must be provided in `toml` format. The example directory also enables connectors for ClickHouse, Delta Lake, Apache Doris, Apache Iceberg, and InfluxDB. Without their backing services (or their compiled plugins) these are reported with the `Error` status, but they don't block the remaining connectors. Set `enabled = false` in their files to skip them entirely. +Run these commands from the root of the same Iggy source checkout used for the server and plugins. This guide targets server 0.9.0, including its edge builds. -2. Run `docker compose up -d` from `/examples/rust/src/sink-data-producer` which will start the Quickwit server to be used by an example sink connector. At this point, you can access the Quickwit UI at [http://localhost:7280](http://localhost:7280) - check this dashboard again later on, after the `events` index will be created. +1. Build the server, CLI, runtime and quick-start plugins: -3. Set environment variable `IGGY_CONNECTORS_CONFIG_PATH=core/connectors/runtime/example_config/config.toml` (adjust the path as needed) pointing to the runtime configuration file. + ```bash + cargo build --release -p server -p iggy-cli -p iggy-connectors \ + -p iggy_connector_random_source -p iggy_connector_stdout_sink \ + -p iggy_connector_quickwit_sink + ``` + + For a debug build, omit `--release` and replace `target/release` with `target/debug` in both the commands and plugin paths. Make sure that the plugins specified in `core/connectors/runtime/example_config/connectors/` directory under `path` are available. The configuration must be provided in `toml` format. + The example directory also enables connectors for ClickHouse, Delta Lake, Apache Doris, Apache Iceberg, and InfluxDB. Without their backing services (or their compiled plugins) these are reported with the `Error` status, but they don't block the remaining connectors. Set `enabled = false` in their files to skip them entirely. + +2. Run `docker compose -f examples/rust/src/sink-data-producer/docker-compose.yml up -d`, which will start the Quickwit server to be used by an example sink connector. At this point, you can access the Quickwit UI at [http://localhost:7280](http://localhost:7280) - check this dashboard again later on, after the `events` index will be created. + +3. In the terminal that will run the connectors, set the runtime configuration path: + + ```bash + export IGGY_CONNECTORS_CONFIG_PATH=core/connectors/runtime/example_config/config.toml + ``` + +4. Start the Iggy server in a separate terminal with credentials matching the sample connector configuration: + + ```bash + IGGY_ROOT_USERNAME=iggy IGGY_ROOT_PASSWORD=iggy cargo run --bin iggy-server --release + ``` -4. Start the Iggy server and invoke the following commands via Iggy CLI to create the example streams and topics used by the sample connectors. + With the server running, create the example streams and topics using the CLI from this checkout. An existing server must have these credentials, or you must adjust the commands and connector configuration to match it. ```bash - iggy --username iggy --password iggy stream create example_stream - iggy --username iggy --password iggy topic create example_stream example_topic 1 none 1d - iggy --username iggy --password iggy stream create qw - iggy --username iggy --password iggy topic create qw records 1 none 1d + target/release/iggy --username iggy --password iggy stream create example_stream + target/release/iggy --username iggy --password iggy topic create example_stream example_topic 1 none 1d + target/release/iggy --username iggy --password iggy stream create qw + target/release/iggy --username iggy --password iggy topic create qw records 1 none 1d ``` -5. Execute `cargo run --example sink-data-producer -r` which will start the example data producer application, sending the messages to previously created `qw` stream and `records` topic (this will be used by the Quickwit sink connector). +5. Execute `cargo run --example sink-data-producer --release`, which sends 100 batches of messages to previously created `qw` stream and `records` topic (this will be used by the Quickwit sink connector). -6. Start the connector runtime `cargo run --bin iggy-connectors -r` - you should be able to browse Quickwit UI with records being constantly added to the `events` index. At the same time, you should see the new messages being added to the `example_stream` stream and `example_topic` topic by the Random source connector - you can use Iggy Web UI to browse the data. The messages will have applied the basic fields transformations. +6. Start the connector runtime `cargo run --bin iggy-connectors --release` in the terminal configured in step 3. The Quickwit sink indexes the produced records in the `events` index. At the same time, you should see the new messages being added to the `example_stream` stream and `example_topic` topic by the Random source connector - you can [start the Iggy Web UI](/docs/web_ui/start) to browse the data. The messages will have applied the basic fields transformations. ## Configuration @@ -94,7 +115,7 @@ IGGY_CONNECTORS_ENV_PATH - path to the .env file for custom environment variable IGGY_CONNECTORS_CONFIG_PATH - path to the connectors runtime configuration file ``` -Any configuration section can be overridden with `IGGY_CONNECTORS_` prefix, followed by the section name and the key name, e.g. `IGGY_CONNECTORS_IGGY_USERNAME`. +Supported scalar fields and indexed list entries can be overridden with the `IGGY_CONNECTORS_` prefix, followed by section and key names joined by underscores, e.g. `IGGY_CONNECTORS_IGGY_USERNAME`. Header and URL-template maps are configured in TOML. Local connector settings use a per-connector prefix, described in the [runtime configuration guide](/docs/connectors/runtime#local-file-provider). ## Runtime @@ -111,13 +132,13 @@ Each sink should have its own, custom configuration, which is passed along with ## Source -Sources are responsible for producing the messages to the configured stream(s) and topic(s). For example, the Random source connector generates random messages that are then sent to the configured stream and topic. +Sources produce messages to an Iggy stream and topic. Configure one `[[streams]]` entry per source instance: the runtime currently retains only the last configured producer. For example, the Random source connector generates random messages that are then sent to the configured stream and topic. Please refer to the **[Source documentation](/docs/connectors/sources/source)** for the details about the configuration and the sample implementation. ## Building the connectors -New connector can be built simply by implementing either `Sink` or `Source` trait. Please check the **[sink](/docs/connectors/sinks/sink)** or **[source](/docs/connectors/sources/source)** documentation, as well as the existing examples under `/sinks` and `/sources` directories. +New connector can be built simply by implementing either `Sink` or `Source` trait. Please check the **[sink](/docs/connectors/sinks/sink)** or **[source](/docs/connectors/sources/source)** documentation, as well as the existing examples under `core/connectors/sinks` and `core/connectors/sources`. ## Transformations diff --git a/content/docs/connectors/observability.mdx b/content/docs/connectors/observability.mdx index 279451156e..2996d7cc09 100644 --- a/content/docs/connectors/observability.mdx +++ b/content/docs/connectors/observability.mdx @@ -7,29 +7,26 @@ The connector runtime exposes its health through logs, Prometheus metrics, a sta ## Logging -The runtime logs via the [tracing](https://docs.rs/tracing/latest/tracing/) crate. The output format is configured in the `[logging]` section: +The runtime logs via the [tracing](https://docs.rs/tracing/latest/tracing/) crate. Merge the runtime configuration snippets on this page into the [minimal runtime configuration](/docs/connectors/runtime#minimal-configuration). The output format is configured in the `[logging]` section: ```toml [logging] format = "text" # "text" (default) or "json" ``` -Two per-connector flags in the connector configuration files add more detail: +Two per-connector flags, both disabled by default, add more detail. Set them at the top level of an existing connector configuration file, before any TOML section headers: - **`verbose`**: logs additional per-batch details for the connector. - **`benchmark`**: emits per-batch timing events on the `iggy_connectors::benchmark` tracing target. ```toml -type = "source" -key = "random" -# ... verbose = false benchmark = true ``` ## Prometheus metrics -Metrics are served by the runtime HTTP API when the `[http.metrics]` section is enabled (the HTTP API itself is on by default at `127.0.0.1:8081`): +Metrics are served by the runtime HTTP API when the `[http.metrics]` section is enabled (the HTTP API itself is on by default at `127.0.0.1:8081`). Metrics are disabled by default; change `enabled` to `true` to expose them: ```toml [http.metrics] @@ -49,21 +46,23 @@ Per-connector counters, labeled with `connector_key` and `connector_type` (`sour - `iggy_connector_messages_produced_total` - messages received from a source plugin's poll. - `iggy_connector_messages_sent_total` - messages sent to Iggy (source). - `iggy_connector_messages_consumed_total` - messages consumed from Iggy (sink). -- `iggy_connector_messages_processed_total` - messages processed and delivered to the sink plugin. -- `iggy_connector_messages_filtered_total` - messages intentionally dropped by transforms. +- `iggy_connector_messages_processed_total` - messages passed to the sink plugin in batches that returned success. A failed batch adds an error and no processed messages; the runtime cannot infer partial delivery inside the plugin. +- `iggy_connector_messages_filtered_total` - messages intentionally dropped by transforms returning `Ok(None)`, excluding transform errors. - `iggy_connector_errors_total` - errors encountered. -Per-batch stage timings are recorded in the `iggy_connector_stage_duration_seconds` histogram, labeled with `connector_key`, `connector_type`, and `stage` (`prepare`, `ffi`, `decode`, `iggy_send`, `state_save`, `total`), with buckets ranging from 50 microseconds to 5 seconds. +Per-batch stage timings are recorded in the `iggy_connector_stage_duration_seconds` histogram independently of the `benchmark` flag, labeled with `connector_key`, `connector_type`, and `stage`. Both connector types record `decode`, `prepare`, and `total`; sinks also record `ffi`, while sources record `iggy_send` and successful checkpoint writes record `state_save`. Finite buckets range from 50 microseconds to 5 seconds. ## Runtime stats `GET /stats` on the HTTP API returns a JSON snapshot of the runtime: version, process ID, CPU and memory usage (cgroup-aware in containers), uptime, the source/sink totals, and a `connectors` array with per-connector details - key, name, type, plugin version, `enabled`, message counters, error count, and the current status. -The connector statuses are `Starting`, `Running`, `Stopping`, `Stopped`, and `Error`. A connector whose plugin fails to load (missing library, bad configuration) is reported with the `Error` status and the failure message, without blocking the remaining connectors. +Memory values are in bytes. `run_time` is elapsed microseconds, and `start_time` is microseconds since the Unix epoch. + +The JSON status values are `starting`, `running`, `stopping`, `stopped`, and `error`. A connector whose plugin fails to load (for example, a missing library) is reported with `error` without blocking the remaining connectors. Retrieve its `last_error` from `GET /sources/{key}` or `GET /sinks/{key}`; `/stats` does not include the failure message. ## OpenTelemetry -The `[telemetry]` section enables OTLP export of logs and traces: +The `[telemetry]` section configures OTLP export of logs and traces. Set `enabled = true` to export to a running collector: ```toml [telemetry] @@ -79,15 +78,20 @@ transport = "grpc" # "grpc" or "http" endpoint = "http://localhost:4317" ``` +For `transport = "http"`, use complete signal URLs, for example `http://localhost:4318/v1/logs` for logs and `http://localhost:4318/v1/traces` for traces. The runtime does not append those paths. + ## State files -Source connectors can persist their position (see the [SDK documentation](/docs/connectors/sdk)) into the directory configured in the `[state]` section: +With the default `file` state backend, source plugins can supply checkpoints (see the [SDK documentation](/docs/connectors/sdk)) for the directory configured in the `[state]` section. An [HTTP state backend](/docs/connectors/runtime#state-storage) is also available: ```toml [state] +storage = "file" path = "local_state" ``` -Each source gets one file named `source_{key}.state` (e.g. `local_state/source_random.state`) holding the raw state bytes - by convention MessagePack produced by the SDK helpers. Sinks have no state files. Their progress lives in Iggy consumer groups. +A source that supplies checkpoint bytes gets a file named `source_{key}.state` (e.g. `local_state/source_random.state`) holding the raw state bytes - by convention MessagePack produced by the SDK helpers. Sinks have no state files. Their progress lives in Iggy consumer groups. + +On Unix, the file backend writes and synchronizes a `.tmp` file, atomically renames it over the previous checkpoint, and synchronizes the parent directory. Newly created files have owner-only permissions (`0600`), since state may carry cursors or tokens. -The write is crash-safe: the state is written and fsynced to a `.tmp` file, atomically renamed over the previous file, and the directory entry is fsynced. Files are created with owner-only permissions (`0600`), since state may carry cursors or tokens. An empty or missing state file means the source starts fresh - deleting a state file resets that source's position. +An empty or missing state file supplies no previous checkpoint when the connector starts. Loading fails if the file cannot be read or its parent directory is unavailable at load time. Stop the runtime before deleting a checkpoint to reset that source on its next start; deleting it while the connector runs does not reset its in-memory position. diff --git a/content/docs/connectors/runtime.mdx b/content/docs/connectors/runtime.mdx index 80cad6e3f0..d89e62523e 100644 --- a/content/docs/connectors/runtime.mdx +++ b/content/docs/connectors/runtime.mdx @@ -4,31 +4,31 @@ description: "How the connector runtime loads plugins, resolves its configuratio --- Runtime is responsible for managing the lifecycle of the connectors and providing the necessary infrastructure for the connectors to run. -The runtime uses a shared [Tokio runtime](https://tokio.rs) to manage the asynchronous tasks and events across all connectors. Additionally, it has built-in support for logging via [tracing](https://docs.rs/tracing/latest/tracing/) crate. +The runtime uses a shared [Tokio runtime](https://tokio.rs) for its connector-management and forwarding tasks. Each loaded plugin library also has an SDK Tokio runtime shared by its instances. Additionally, it has built-in support for logging via [tracing](https://docs.rs/tracing/latest/tracing/) crate. The connector are implemented as Rust libraries, and these are loaded dynamically during the runtime initialization process. -Internally, [dlopen2](https://github.com/OpenByteDev/dlopen2) provides a safe and efficient way of loading the plugins via C FFI. +Internally, [dlopen2](https://github.com/OpenByteDev/dlopen2) loads plugin libraries and resolves their C FFI symbols. Plugins execute inside the runtime process. By default, runtime will look for the configuration file, to decide which connectors to load and how to configure them. -To start the connector runtime, simply run `cargo run --bin iggy-connectors`. +Set the broker credentials and connector configuration directory before starting the runtime. The embedded default has an empty connector directory and cannot start unchanged. For a complete setup, follow the [quick start](/docs/connectors/introduction#quick-start). -The [docker image](https://hub.docker.com/r/apache/iggy-connect) is available, and can be fetched via `docker pull apache/iggy-connect`. +The [docker image](https://hub.docker.com/r/apache/iggy-connect) is available, and can be fetched via `docker pull apache/iggy-connect:edge`. ## How configuration is resolved - The configuration file path defaults to `core/connectors/runtime/config.toml` and can be overridden by the `IGGY_CONNECTORS_CONFIG_PATH` environment variable. -- A default configuration is embedded in the binary and always merged as the base, so every section has sane defaults even when the file omits it. -- On startup the runtime loads environment variables from a `.env` file in the working directory, or from the file pointed to by `IGGY_CONNECTORS_ENV_PATH`. -- Each configuration key can be additionally overridden by an environment variable using the `IGGY_CONNECTORS_
_` convention (nested keys joined by underscores), e.g. `IGGY_CONNECTORS_IGGY_USERNAME` or `IGGY_CONNECTORS_HTTP_ADDRESS`. +- A default configuration is embedded in the binary and always merged as the base, so omitted settings inherit its values. A local provider still needs a nonempty `connectors.config_dir`. +- On startup the runtime loads environment variables from the first `.env` file found in the working directory or its parents, or from the file pointed to by `IGGY_CONNECTORS_ENV_PATH`. +- Supported scalar fields and indexed list entries can be overridden by environment variables using the `IGGY_CONNECTORS_
_` convention (nested keys joined by underscores), e.g. `IGGY_CONNECTORS_IGGY_USERNAME` or `IGGY_CONNECTORS_HTTP_ADDRESS`. ## How plugins are resolved The `path` field of a connector configuration accepts both `plugin.so` and `plugin` - the OS-specific extension (`.so` / `.dylib` / `.dll`) is appended when missing. Absolute paths are checked at the literal location. Relative paths are searched in order: 1. Literal path (relative to the working directory) -2. Directory of the runtime binary +2. Directory of the runtime binary (filename only) 3. Current working directory (filename only) 4. `/usr/lib`, `/usr/lib64`, `/lib`, `/lib64`, `/usr/local/lib`, `/usr/local/lib64` @@ -36,7 +36,7 @@ A connector whose plugin cannot be resolved or loaded is surfaced with the `Erro ## Minimal configuration -The minimal viable configuration requires at least the Iggy credentials to create 2 separate instances of producer & consumer connections, the state directory path where source connectors can store their optional state, and the connectors configuration provider settings. +The runtime opens two Iggy TCP clients, one for producers and one for consumers. Set credentials matching the broker and a connector configuration provider. Omitted settings use the embedded defaults, including file-based source state storage. Save this example as `connectors.toml` in the repository root and replace `path/to/connectors` with your connector configuration directory. ```toml [iggy] @@ -58,8 +58,54 @@ config_type = "local" config_dir = "path/to/connectors" ``` +Start it from the repository root: + +```bash +IGGY_CONNECTORS_CONFIG_PATH=connectors.toml cargo run --bin iggy-connectors +``` + Beyond these sections, the runtime configuration also supports `[http]` for the HTTP API (see below), and `[telemetry]`, `[logging]`, and `[http.metrics]` covered on the **[Observability page](/docs/connectors/observability)**. +## State storage + +Source plugins supply optional checkpoint bytes. The default `file` backend writes them to `{state.path}/source_{key}.state` using a temporary file, file synchronization and atomic rename. On Unix, it also synchronizes the parent directory. The runtime stores these bytes unchanged; the SDK provides MessagePack serialization helpers. + +To use an HTTP state server, replace the `[state]` section above and add: + +```toml +[state] +storage = "http" + +[state.http] +url = "http://127.0.0.1:8080/connectors/state" +load_method = "get" +save_method = "put" +timeout = "5s" + +[state.http.request_headers] +authorization = "Bearer your-state-api-token" + +[state.http.retry] +enabled = true +max_attempts = 4 +initial_backoff = "200ms" +max_backoff = "2s" +backoff_multiplier = 2 +``` + +The runtime appends `source_{key}` as a URL path segment, preserving any query string. URL query values are redacted from runtime configuration logs and transport errors. Static request headers are passed to the state server; configure this map in TOML. The other state settings support environment overrides such as `IGGY_CONNECTORS_STATE_STORAGE` and `IGGY_CONNECTORS_STATE_HTTP_URL`. + +The HTTP server must implement this checkpoint contract: + +- Load uses `GET` by default, or `POST`. Return `200` with the stored bytes and a strong `ETag`, or `404` when no state exists. An empty `200` body is valid state. +- Save uses `PUT` by default, or `POST`/`PATCH`, with `Content-Type: application/octet-stream`. Atomically enforce `If-Match` for a known ETag or `If-None-Match: *` for a new checkpoint. Successful `200`, `201` or `204` responses must include the new strong `ETag`. +- Each logical save has an `Idempotency-Key`, reused across retries and resolution of an uncertain write. The server must replay the original outcome for that key without applying the write again. +- `If-Match`, `If-None-Match`, `Idempotency-Key` and `Content-Type` are managed by the runtime and cannot be overridden in `request_headers`. + +Timeouts, connection failures, `425`, `429` and `5xx` responses are retried. `max_attempts` counts retries after the first request, so the default 4 allows up to 5 requests. Integer-seconds `Retry-After` values are honored up to `max_backoff`. Other save failures, including version conflicts or missing ETags, latch the provider: further saves fail until the connector restarts. An HTTP state-load failure while an enabled source starts aborts runtime startup. + +The runtime forwards a source batch to Iggy, saves its optional checkpoint, then acknowledges the batch to the plugin. A failed send or checkpoint save produces a negative acknowledgement. Remote checkpoint durability depends on the state server. Connector lifecycle operations do not delete remote state. + ## Configuration Providers The runtime supports two types of configuration providers for managing connector configurations: @@ -76,13 +122,13 @@ config_dir = "path/to/connectors" Additional mechanics of the local provider: -- Only `*.toml` files are loaded - hidden files (starting with `.`) and `Cargo.toml` are skipped. -- Multiple files sharing the same connector `key` form a version history (each file carries its own `version`). The active version of each connector is persisted in `{config_dir}/.active_versions.toml` and can be switched via the HTTP API. -- Individual `plugin_config` fields can be overridden per connector via environment variables using the `IGGY_CONNECTORS_{SINK|SOURCE}_{KEY}_PLUGIN_CONFIG_` convention, e.g. `IGGY_CONNECTORS_SINK_QUICKWIT_PLUGIN_CONFIG_URL`. +- Only `*.toml` files directly inside the directory are loaded - hidden files (starting with `.`) and `Cargo.toml` are skipped. +- Multiple files sharing the same connector `key` form a version history (each file carries its own `version`). The active version of each connector is persisted in `{config_dir}/.active_versions.toml` and can be selected via the HTTP API for the next process startup. Without a saved selection, the highest version is used. Changing this selection does not update a running connector. The connector restart endpoint currently loads the highest local version, which can differ from the selected active version. +- Top-level `plugin_config` fields can be overridden per connector via environment variables using the `IGGY_CONNECTORS_{SINK|SOURCE}_{KEY}_PLUGIN_CONFIG_` convention, e.g. `IGGY_CONNECTORS_SINK_QUICKWIT_PLUGIN_CONFIG_URL`. ### HTTP Configuration Provider -The HTTP configuration provider allows the runtime to fetch connector configurations from a remote HTTP/REST API. This enables centralized configuration management and dynamic configuration updates. +The HTTP configuration provider allows the runtime to fetch connector configurations from a remote HTTP/REST API. The provider fetches active configurations at startup and handles configuration operations requested through the runtime API. It does not periodically poll for remote changes. ```toml [connectors] @@ -95,7 +141,7 @@ api-key = "your-api-key" [connectors.retry] enabled = true -max_attempts = 3 +max_attempts = 3 # Retries after the first request, up to four requests total initial_backoff = "1 s" max_backoff = "30 s" backoff_multiplier = 2 @@ -119,8 +165,8 @@ error_path = "error" # Path to error in response (e.g., {"error": "..."}) - **timeout** (optional): HTTP request timeout (default: 10s) - **request_headers** (optional): Custom headers to include in all HTTP requests (e.g., authentication headers) - **url_templates** (optional): Custom URL templates for API endpoints. Supports variable substitution with `{key}` and `{version}` placeholders. -- **response.data_path** (optional): JSON path to extract response data from nested structures (e.g., "data.config") -- **response.error_path** (optional): JSON path to check for errors in responses +- **response.data_path** (optional): Dot-separated object keys or numeric array indexes used to extract data (e.g., `data.config` or `data.0`). +- **response.error_path** (optional): A path with the same syntax. Any non-null value at this path is treated as an error, including `false` or an empty string. #### Default URL Templates @@ -147,7 +193,7 @@ The HTTP provider expects the remote API to implement these endpoints and return Connector runtime has an HTTP API which is **enabled by default** at `127.0.0.1:8081`. It can be disabled by setting the `enabled` flag to `false` in the `[http]` section. -When `api_key` is set, every request must carry it in the `api-key` header - except `GET /` and `GET /health`, which stay public. +When `api_key` is set, every request must carry it in the `api-key` header - except `GET /` and `GET /health`, which stay public. An empty key disables authentication. Configuration endpoints return plugin credentials and accept configuration changes, so set an API key before exposing this API to other hosts. Enable TLS or use a trusted TLS proxy for remote access. ```toml [http] # Optional HTTP API configuration @@ -177,14 +223,14 @@ key_file = "core/certs/iggy_key.pem" Currently, it does expose the following endpoints: - `GET /`: welcome message. -- `GET /health`: health status of the runtime. +- `GET /health`: process liveness response. It does not check connector health; inspect `/stats`, `/sources` or `/sinks` for connector status. - `GET /stats`: runtime statistics (process info plus per-connector status and message counters). - `GET /metrics`: Prometheus metrics, available when `[http.metrics]` is enabled (path configurable via its `endpoint` key). - `GET /sinks`: list of sinks. - `GET /sinks/{key}`: sink details. - `GET /sinks/{key}/configs`: list of configuration versions for the sink. - `POST /sinks/{key}/configs`: add a new configuration version for the sink. -- `DELETE /sinks/{key}/configs`: delete the sink configuration. +- `DELETE /sinks/{key}/configs`: delete one configuration version, chosen by the `version` query parameter or the locally saved active selection. - `GET /sinks/{key}/configs/{version}`: configuration details for a specific version. - `GET /sinks/{key}/configs/active`: active configuration details. - `PUT /sinks/{key}/configs/active`: activate a specific configuration version for the sink. @@ -195,7 +241,7 @@ Currently, it does expose the following endpoints: - `GET /sources/{key}`: source details. - `GET /sources/{key}/configs`: list of configuration versions for the source. - `POST /sources/{key}/configs`: add a new configuration version for the source. -- `DELETE /sources/{key}/configs`: delete the source configuration. +- `DELETE /sources/{key}/configs`: delete one configuration version, chosen by the `version` query parameter or the locally saved active selection. - `GET /sources/{key}/configs/{version}`: configuration details for a specific version. - `GET /sources/{key}/configs/active`: active configuration details. - `PUT /sources/{key}/configs/active`: activate a specific configuration version for the source. diff --git a/content/docs/connectors/sdk.mdx b/content/docs/connectors/sdk.mdx index 34091fa4aa..51a1d6e44d 100644 --- a/content/docs/connectors/sdk.mdx +++ b/content/docs/connectors/sdk.mdx @@ -12,10 +12,19 @@ Moreover, it contains both, the `decoders` and `encoders` modules, implementing A source produces messages to the configured stream and topic. A sink consumes messages from the configured stream(s) and topic(s): ```rust +use async_trait::async_trait; +use iggy_connector_sdk::{ + ConsumedMessage, Error, MessagesMetadata, ProducedMessages, TopicMetadata, + source::SourceBatchResult, +}; + #[async_trait] pub trait Source: Send + Sync { async fn open(&mut self) -> Result<(), Error>; async fn poll(&self) -> Result; + async fn on_batch_result(&self, _result: SourceBatchResult) -> Result<(), Error> { + Ok(()) + } async fn close(&mut self) -> Result<(), Error>; } @@ -32,29 +41,17 @@ pub trait Sink: Send + Sync { } ``` -Expose the implementation to the runtime with the matching macro, and provide the expected `new()` constructor: - -```rust -sink_connector!(MySink); - -impl MySink { - pub fn new(id: u32, config: MySinkConfig) -> Self { /* ... */ } -} -``` +Expose the implementation with `sink_connector!(MySink)` or `source_connector!(MySource)`. The sink constructor has the signature `new(id: u32, config: MySinkConfig) -> Self`; the source constructor also takes `state: Option`. The **[sink guide](/docs/connectors/sinks/sink)** and **[source guide](/docs/connectors/sources/source)** provide complete plugin implementations and build setup. -```rust -source_connector!(MySource); +The macros generate the C FFI symbols loaded by the runtime: `iggy_sink_open`, `iggy_sink_consume`, `iggy_sink_close`, and `iggy_sink_version` for sinks, and `iggy_source_open`, `iggy_source_handle_v2`, `iggy_source_batch_result`, `iggy_source_close`, and `iggy_source_version` for sources. Configurations cross FFI as JSON, message batches and metadata use postcard, and source checkpoint bytes are opaque. The SDK creates one Tokio runtime per loaded plugin library, shared by that library's connector instances. -impl MySource { - pub fn new(id: u32, config: MySourceConfig, state: Option) -> Self { /* ... */ } -} -``` +## State -The macros generate the C FFI symbols loaded by the runtime: `iggy_sink_open`, `iggy_sink_consume`, `iggy_sink_close`, and `iggy_sink_version` for sinks, and `iggy_source_open`, `iggy_source_handle`, `iggy_source_close`, and `iggy_source_version` for sources. Data crossing the FFI boundary is serialized using postcard, and each plugin receives its own Tokio runtime inside the SDK. +Source connectors can persist their position between restarts. Each batch of `ProducedMessages` may carry an optional `ConnectorState`. After processing and sending the batch to Iggy, the runtime saves that state through the configured file or HTTP backend before acknowledging the batch. The last saved state is passed to `new()` on the next startup. `ConnectorState` wraps raw bytes and provides `serialize`/`deserialize` helpers backed by MessagePack. -## State +Sources with cursors or destructive operations must stage changes during `poll()` and apply them on `SourceBatchResult::Ack`; `Nack` discards staged progress so the batch can be polled again. The SDK allows one batch in flight and stops polling if `on_batch_result()` returns an error. Its default no-op is suitable only for sources without staged work. -Source connectors can persist their position between restarts. Each batch of `ProducedMessages` may carry an optional `ConnectorState` - the runtime saves it to a file after the batch is sent to Iggy, and passes the last saved state to `new()` on the next startup. `ConnectorState` wraps raw bytes and provides `serialize`/`deserialize` helpers backed by MessagePack. Sinks don't manage state. Their offsets are tracked by Iggy consumer groups. See the **[Observability page](/docs/connectors/observability)** for where and how state files are stored. +Sinks can keep in-memory state, but the runtime does not persist it. Their consumer-group offsets are tracked by Iggy. See the **[Observability page](/docs/connectors/observability)** for checkpoint storage and the **[sink guide](/docs/connectors/sinks/sink)** for offset handling. ## Retry utilities @@ -81,9 +78,16 @@ The SDK includes support for Protocol Buffers (protobuf) format with both encodi ### Configuration Example -Here's a complete example configuration for using Protocol Buffers with Iggy connectors. No protobuf-specific plugin ships with the runtime, so the example assumes a custom source and sink you implement yourself. The `schema = "proto"` stream setting and the `proto_convert` transform work with any connector. +This example uses the Random source and Stdout sink from the matching checkout. Start a server using the **[getting-started guide](/docs/introduction/getting-started)**, then build the plugins, runtime, and CLI from the repository root: -**Main runtime config (config.toml):** +```bash +cargo build --release -p iggy_connector_random_source -p iggy_connector_stdout_sink -p iggy-connectors -p iggy-cli +mkdir -p connectors +``` + +The source's `schema = "proto"` selects the default protobuf encoder, which wraps each JSON record in a `google.protobuf.StringValue` inside `google.protobuf.Any`. The sink reads raw bytes and applies `proto_convert` to expose the Any envelope as JSON. This does not unpack a custom protobuf message schema. + +**Main runtime config (connectors.toml):** ```toml [iggy] @@ -93,7 +97,7 @@ password = "iggy" [connectors] config_type = "local" -config_dir = "path/to/connectors" +config_dir = "connectors" ``` **Source connector config (connectors/protobuf_source.toml):** @@ -104,7 +108,7 @@ key = "protobuf" enabled = true version = 0 name = "Protobuf Source" -path = "target/release/libiggy_connector_protobuf_source" +path = "target/release/libiggy_connector_random_source" [[streams]] stream = "protobuf_stream" @@ -114,9 +118,10 @@ batch_length = 1000 linger_time = "5ms" [plugin_config] -schema_path = "schemas/message.proto" -message_type = "com.example.Message" -use_any_wrapper = true +interval = "100ms" +messages_range = [1, 10] +payload_size = 32 +max_count = 100 ``` **Sink connector config (connectors/protobuf_sink.toml):** @@ -127,12 +132,15 @@ key = "protobuf" enabled = true version = 0 name = "Protobuf Sink" -path = "target/release/libiggy_connector_protobuf_sink" +path = "target/release/libiggy_connector_stdout_sink" [[streams]] stream = "protobuf_stream" topics = ["protobuf_topic"] -schema = "proto" +schema = "raw" + +[plugin_config] +print_payload = true [transforms.proto_convert] enabled = true @@ -140,7 +148,6 @@ source_format = "proto" target_format = "json" include_paths = ["."] preserve_unknown_fields = false -field_mappings = { "old_field" = "new_field", "legacy_id" = "id" } [transforms.proto_convert.conversion_options] validate_messages = true @@ -150,116 +157,199 @@ type_url_prefix = "type.googleapis.com" strict_mode = false ``` -The format-conversion transforms define no per-key defaults. Every non-optional key shown above must be present, or the configuration fails to deserialize (`schema_path`, `message_type`, `field_mappings`, and `descriptor_set` are the optional ones). +Create the stream and topic, then start the runtime from the repository root: + +```bash +./target/release/iggy --username iggy --password iggy stream create protobuf_stream +./target/release/iggy --username iggy --password iggy topic create protobuf_stream protobuf_topic 1 none 1d +IGGY_CONNECTORS_CONFIG_PATH=connectors.toml ./target/release/iggy-connectors +``` + +The source sends 100 records, then continues polling without new messages. Stdout logs message offsets and the serialized JSON envelope bytes, containing `type_url` and base64 `value`. The sink's `raw` schema also determines how the plugin receives those transformed bytes. + +The format-conversion transforms define no per-key defaults. Every non-optional key shown above must be present, or the configuration fails to deserialize (`schema_path`, `message_type`, `field_mappings`, and `descriptor_set` are optional). The two `[[streams]]` shapes differ: a source produces to a single `topic` and can tune batching via `batch_length` and `linger_time`, while a sink consumes from a list of `topics` and can additionally set `batch_length`, `poll_interval`, and `consumer_group`. ### Key Configuration Options -#### Source Configuration +#### Programmatic Encoder and Decoder Configuration + +These are SDK configuration fields, not Random or Stdout `plugin_config` keys. The runtime's `schema = "proto"` uses the default encoder or decoder. - **`schema_path`**: Path to the `.proto` file containing message definitions - **`message_type`**: Fully qualified name of the protobuf message type to use -- **`use_any_wrapper`**: Whether to wrap messages in `google.protobuf.Any` for type safety +- **`use_any_wrapper`**: Selects the Any fallback when no message descriptor is loaded; a loaded descriptor takes precedence #### Transform Options - **`proto_convert`**: Transform for converting between protobuf and other formats - **`source_format`** / **`target_format`**: Formats to convert between - any schema value (`json`, `raw`, `text`, `proto`, `flat_buffer`, `avro`) -- **`preserve_unknown_fields`**: Whether to keep fields that are not present in the schema during conversion +- **`preserve_unknown_fields`**: Accepted by `proto_convert`, but currently has no effect - **`include_paths`**: Additional directories searched for imported `.proto` files -- **`field_mappings`**: Mapping of field names for transformation (e.g., `"old_field" = "new_field"`) -- **`conversion_options`**: Fine-tuning knobs: `validate_messages`, `pretty_json`, `include_metadata`, `type_url_prefix`, `strict_mode` +- **`field_mappings`**: Renames fields in a JSON input object before conversion (e.g., `"old_field" = "new_field"`) +- **`conversion_options`**: `pretty_json` controls JSON text output and `include_metadata` enriches supported protobuf-to-JSON paths. `validate_messages`, `type_url_prefix`, and `strict_mode` are accepted but currently have no effect The `schema_registry_url` field is reserved and currently not implemented. The SDK never contacts a schema registry, and schemas are loaded only from `schema_path` or `descriptor_set`. ### Supported Features -- **Encoding**: Convert JSON, Text, and Raw data to protobuf format -- **Decoding**: Parse protobuf messages into JSON format with type information -- **Transforms**: Convert between protobuf and other formats -- **Field Mapping**: Transform field names during format conversion -- **Any Wrapper**: Support for `google.protobuf.Any` message wrapper +- **Encoding**: A loaded message descriptor encodes matching JSON fields. The encoder supports booleans, strings, all protobuf integer types, and base64 strings for bytes or already-encoded nested messages. Float, double, and enum fields are unsupported by the encoder; nested JSON objects, repeated fields, maps, and proto2 groups are not a general-purpose schema conversion path. +- **Decoding**: A loaded descriptor extracts present fields. Integer, boolean, and string fields become JSON values; bytes become base64 and nested messages become metadata with base64 content. Missing fields are not filled with protobuf defaults, and float/double fields produce `unsupported_wire_type` placeholders. Without a descriptor, the default decoder returns an Any envelope's `type_url` and base64 `value`. +- **Transforms**: `proto_convert` supports JSON-to-protobuf schema encoding for scalar fields, including floating-point numbers and numeric enum values; bytes and already-encoded nested messages use base64 strings. It logs and omits fields it cannot encode. Its protobuf-to-JSON path exposes Any metadata or raw-data metadata rather than decoding a custom message descriptor. Converting protobuf to `flat_buffer` or `avro` rewraps bytes without transcoding them. +- **Field Mapping**: Encoder/decoder mappings use protobuf field names as keys and JSON field names as values. The encoder applies that mapping in reverse. Transform mappings rename JSON input keys directly. +- **Any Wrapper**: The default encoder puts JSON/text in a `google.protobuf.StringValue`, or binary data in a `google.protobuf.BytesValue`, inside `google.protobuf.Any`. The default decoder exposes the envelope without unpacking its inner message. ### Programmatic Usage +From the matching repository root, create an example crate and schema directory: + +```bash +mkdir -p connector-sdk-example/src schemas +``` + +Save this as `connector-sdk-example/Cargo.toml`. The path dependency uses the SDK from the same checkout as the runtime: + +```toml +[package] +name = "connector-sdk-example" +version = "0.1.0" +edition = "2024" + +[dependencies] +iggy_connector_sdk = { path = "../core/connectors/sdk" } +simd-json = { version = "0.18.1", features = ["serde_impl"] } + +[workspace] +``` + +Save this as `schemas/user.proto`: + +```protobuf +syntax = "proto3"; +package com.example; + +message User { + uint64 id = 1; + string name = 2; +} +``` + +Each Rust example below is a complete `connector-sdk-example/src/main.rs`. Run it from the repository root so the relative schema path resolves: + +```bash +cargo run --manifest-path connector-sdk-example/Cargo.toml +``` + #### Dynamic Schema Loading You can load or reload schemas programmatically: ```rust -use iggy_connector_sdk::decoders::proto::{ProtoStreamDecoder, ProtoConfig}; +use iggy_connector_sdk::decoders::proto::{ProtoConfig, ProtoStreamDecoder}; +use iggy_connector_sdk::encoders::proto::{ProtoEncoderConfig, ProtoStreamEncoder}; +use iggy_connector_sdk::{Error, Payload, StreamDecoder, StreamEncoder}; use std::path::PathBuf; -let mut decoder = ProtoStreamDecoder::new(ProtoConfig { - schema_path: None, - use_any_wrapper: true, - ..Default::default() -}); - -let config_with_schema = ProtoConfig { - schema_path: Some(PathBuf::from("schemas/user.proto")), - message_type: Some("com.example.User".to_string()), - ..Default::default() -}; - -match decoder.update_config(config_with_schema, true) { - Ok(()) => println!("Schema loaded successfully"), - Err(e) => eprintln!("Failed to load schema: {}", e), +fn main() -> Result<(), Error> { + let mut decoder = ProtoStreamDecoder::new_default(); + decoder.update_config( + ProtoConfig { + schema_path: Some(PathBuf::from("schemas/user.proto")), + message_type: Some("com.example.User".to_string()), + ..ProtoConfig::default() + }, + true, + )?; + let encoder = ProtoStreamEncoder::new_with_config(ProtoEncoderConfig { + schema_path: Some(PathBuf::from("schemas/user.proto")), + message_type: Some("com.example.User".to_string()), + ..ProtoEncoderConfig::default() + }); + let encoded = encoder.encode(Payload::Json(simd_json::json!({ + "id": 1, + "name": "Alice" + })))?; + println!("{}", decoder.decode(encoded)?); + Ok(()) } ``` The encoder follows the same pattern: ```rust -use iggy_connector_sdk::encoders::proto::{ProtoStreamEncoder, ProtoEncoderConfig}; +use iggy_connector_sdk::encoders::proto::{ProtoEncoderConfig, ProtoStreamEncoder}; +use iggy_connector_sdk::{Error, Payload, StreamEncoder}; use std::path::PathBuf; -let mut encoder = ProtoStreamEncoder::new_with_config(ProtoEncoderConfig { - schema_path: Some(PathBuf::from("schemas/event.proto")), - message_type: Some("com.example.Event".to_string()), - use_any_wrapper: false, - ..Default::default() -}); - -if let Err(e) = encoder.load_schema() { - eprintln!("Schema reload failed: {}", e); +fn main() -> Result<(), Error> { + let mut encoder = ProtoStreamEncoder::new_with_config(ProtoEncoderConfig { + schema_path: Some(PathBuf::from("schemas/user.proto")), + message_type: Some("com.example.User".to_string()), + use_any_wrapper: false, + ..ProtoEncoderConfig::default() + }); + encoder.load_schema()?; + let encoded = encoder.encode(Payload::Json(simd_json::json!({ + "id": 1, + "name": "Alice" + })))?; + println!("{encoded:?}"); + Ok(()) } ``` #### Creating Converters with Schema +The loaded schema is used for JSON-to-protobuf conversion. This example maps `user_id` and `full_name` to the schema's field names: + ```rust -use iggy_connector_sdk::transforms::proto_convert::{ProtoConvert, ProtoConvertConfig}; -use iggy_connector_sdk::Schema; +use iggy_connector_sdk::transforms::{ProtoConvert, ProtoConvertConfig, Transform}; +use iggy_connector_sdk::{DecodedMessage, Error, Payload, Schema, TopicMetadata}; use std::collections::HashMap; use std::path::PathBuf; -let converter = ProtoConvert::new(ProtoConvertConfig { - source_format: Schema::Proto, - target_format: Schema::Json, - schema_path: Some(PathBuf::from("schemas/user.proto")), - message_type: Some("com.example.User".to_string()), - field_mappings: Some(HashMap::from([ - ("user_id".to_string(), "id".to_string()), - ("full_name".to_string(), "name".to_string()), - ])), - ..ProtoConvertConfig::default() -}); - -let mut converter_with_manual_loading = ProtoConvert::new(ProtoConvertConfig::default()); -if let Err(e) = converter_with_manual_loading.load_schema() { - eprintln!("Manual schema loading failed: {}", e); +fn main() -> Result<(), Error> { + let converter = ProtoConvert::new(ProtoConvertConfig { + source_format: Schema::Json, + target_format: Schema::Proto, + schema_path: Some(PathBuf::from("schemas/user.proto")), + message_type: Some("com.example.User".to_string()), + field_mappings: Some(HashMap::from([ + ("user_id".to_string(), "id".to_string()), + ("full_name".to_string(), "name".to_string()), + ])), + ..ProtoConvertConfig::default() + }); + let metadata = TopicMetadata { + stream: "users".to_string(), + topic: "users".to_string(), + }; + let message = DecodedMessage { + id: None, + offset: None, + checksum: None, + timestamp: None, + origin_timestamp: None, + headers: None, + payload: Payload::Json(simd_json::json!({ + "user_id": 1, + "full_name": "Alice" + })), + }; + if let Some(converted) = converter.transform(&metadata, message)? { + println!("{:?}", converted.payload); + } + Ok(()) } ``` ### Usage Notes -- **Automatic Loading**: Schemas are loaded automatically when `schema_path` or `descriptor_set` is provided in config -- **Manual Loading**: Use `load_schema()` method for dynamic schema loading or reloading -- **Error Handling**: Schema loading errors are handled gracefully with fallback to Any wrapper mode -- **Immutable Design**: Converters are created with fixed configuration - create new instances for different schemas -- When `use_any_wrapper` is enabled, messages are wrapped in `google.protobuf.Any` for better type safety -- The `proto_convert` transform can be used to convert protobuf messages to JSON for easier processing -- Field mappings allow you to rename fields during format conversion +- **Automatic Loading**: Constructors attempt to load `schema_path` or `descriptor_set`; `schema_path` takes precedence when both are set. Constructors log loading errors and return an instance without a loaded schema. +- **Manual Loading**: `load_schema()` reloads the configured source. A returned error preserves an already-loaded schema. `update_config(config, true)` also restores the previous configuration on error; `false` changes the configuration while retaining the cached schema. +- **Fallbacks**: Missing files, compilation failures, absent schema sources, or an unmatched `message_type` can return `Ok(())` without an active message descriptor. Successful reloads into fallback mode clear the previous descriptor. Invalid protobuf syntax and malformed descriptor bytes return errors. Check the actual encoded/decoded result when validating a schema setup. +- **Encoding Errors**: Errors encoding a loaded message descriptor are returned to the caller. The encoder does not retry that message as Any. The decoder attempts Any after a schema decoding error. +- **Transform Configuration**: Create a new converter to change its configuration. `load_schema()` can reload its existing source. Without a descriptor, JSON-to-protobuf conversion produces JSON text in `Payload::Proto`, not a schema-encoded binary message. +- **Format Options**: Encoder `preserve_unknown_fields`, `compact_encoding`, `validate_message`, and `deterministic_encoding` are accepted but have no effect. Decoder `preserve_unknown_fields` retains unknown varints as numbers and length-delimited data as base64; fixed-width unknown fields become placeholders. It does not retain the original wire encoding. See the [Transforms page](/docs/connectors/transforms) for conversion-specific limits. - Protocol Buffers provide efficient binary serialization compared to JSON diff --git a/content/docs/connectors/sinks/clickhouse.mdx b/content/docs/connectors/sinks/clickhouse.mdx index 0d7a141a7a..bf8ccf7d21 100644 --- a/content/docs/connectors/sinks/clickhouse.mdx +++ b/content/docs/connectors/sinks/clickhouse.mdx @@ -9,6 +9,20 @@ This page is a curated subset of the documentation. The canonical reference is t ## Configuration +Use the broker credentials and runtime setup from the [sink guide](/docs/connectors/sinks/sink#configuration). From the matching 0.9.0/edge repository root, build the plugin: + +```bash +cargo build --release -p iggy_connector_clickhouse_sink +``` + +The sample expects a local ClickHouse HTTP endpoint with user `default` and an empty password. Create a test table with matching columns: + +```bash +curl --fail-with-body http://localhost:8123/ --data-binary 'CREATE TABLE events (id UInt64, name String) ENGINE = MergeTree ORDER BY id' +``` + +Save the configuration below in the runtime's connector directory. Set the endpoint, credentials, database, and table for your deployment. + ```toml type = "sink" key = "clickhouse" @@ -16,7 +30,6 @@ enabled = true version = 0 name = "ClickHouse sink" path = "target/release/libiggy_connector_clickhouse_sink" -plugin_config_format = "toml" [[streams]] stream = "example_stream" @@ -39,6 +52,14 @@ retry_delay = 1 # seconds verbose_logging = false ``` +Create the Iggy stream and topic, then send a JSON row before starting the runtime: + +```bash +./target/release/iggy --username iggy --password iggy stream create example_stream +./target/release/iggy --username iggy --password iggy topic create example_stream example_topic 1 none 1d +./target/release/iggy --username iggy --password iggy message send --partition-id 0 example_stream example_topic '{"id":1,"name":"hello"}' +``` + ### Options | Option | Type | Default | Description | @@ -50,25 +71,28 @@ verbose_logging = false | `password` | string | `""` | ClickHouse password | | `insert_format` | string | `json_each_row` | `json_each_row`, `row_binary`, or `string` | | `string_format` | string | `json_each_row` | ClickHouse format for `string` mode: `json_each_row`, `csv`, or `tsv` | -| `timeout_seconds` | u64 | `30` | HTTP request timeout | -| `max_retries` | u32 | `3` | Total insert attempts on transient errors | -| `retry_delay` | u64 | `1` | Delay between retries, in seconds | +| `timeout_seconds` | u64 | `30` | HTTP request timeout in seconds | +| `max_retries` | u32 | `3` | Total attempts for startup requests and transient insert errors; at least one even when `0` | +| `retry_delay` | u64 | `1` | Base for exponential retry delay, in seconds | | `verbose_logging` | bool | `false` | Log inserts at info level instead of debug | ## Insert Formats -- **`json_each_row`** (default): accepts JSON payloads. Each message is sent as one JSON object per line using ClickHouse's `JSONEachRow` format. ClickHouse coerces JSON values to the column types, so the table can have any schema. -- **`row_binary`**: accepts JSON payloads and serializes them to ClickHouse's `RowBinaryWithDefaults` binary format, which is more efficient than JSON at high volume. The table must already exist. Its schema is fetched from `system.columns` and validated at startup. Requires ClickHouse 23.7 or newer. Columns with a `DEFAULT` expression may be omitted from messages. `MATERIALIZED`, `ALIAS`, and `EPHEMERAL` columns are dropped from the schema. Unsupported column types (128/256-bit integers, `Variant`, native `JSON`, geo types) cause startup to fail. -- **`string`**: accepts text payloads and passes them through unchanged. Use `string_format` to tell ClickHouse which format the payload is in (`csv`, `tsv`, or `json_each_row`). +- **`json_each_row`** (default): accepts JSON payloads. Each payload is serialized on its own line using ClickHouse's `JSONEachRow` format. Send JSON objects whose fields and values are compatible with the existing table and its ClickHouse input settings. The connector does not validate JSON rows against the table schema before sending them. +- **`row_binary`**: accepts JSON payloads and serializes them to ClickHouse's `RowBinaryWithDefaults` binary format, which is more efficient than JSON at high volume. The table must already exist. Its schema is fetched from `system.columns` and validated at startup. Requires [ClickHouse 23.7 or newer](https://presentations.clickhouse.com/2023-release-23.7/index.html), which introduced this format. Columns with a `DEFAULT` expression may be omitted from messages. An explicit JSON `null` requires a nullable column and is stored as `NULL`, even when that column has a default. Missing columns without defaults must be nullable. `MATERIALIZED`, `ALIAS`, and `EPHEMERAL` columns are dropped from the schema. Unsupported column types (128/256-bit integers, `Variant`, native `JSON`, geo types) cause startup to fail. +- **`string`**: accepts text payloads and appends a newline to each payload that does not already end with one. Set the stream `schema = "text"`. Use `string_format` to tell ClickHouse which format the payload is in (`csv`, `tsv`, or `json_each_row`). -In `row_binary` mode the schema is captured once at startup and the insert stream is positional. An `ALTER TABLE` on the target while the connector runs can silently corrupt inserted rows, so restart the connector after any schema change. The self-describing `json_each_row` format (including `string` mode with `string_format = "json_each_row"`) maps values by field name and tolerates schema changes. `string` passthrough with `csv` or `tsv` sends plain positional ClickHouse formats mapped by the current table column order. +In `row_binary` mode the schema is captured once at startup and the insert stream is positional. An `ALTER TABLE` on the target while the connector runs can silently corrupt inserted rows, so restart the connector after any schema change. The self-describing `json_each_row` format (including `string` mode with `string_format = "json_each_row"`) maps values by field name, but changed column names, types, or constraints can still make inserts fail. `string` passthrough with `csv` or `tsv` sends plain positional ClickHouse formats mapped by the current table column order. ## Error Handling & Delivery Semantics -Failed inserts are attempted up to `max_retries` total times with exponential backoff and full jitter, starting from `retry_delay`. A message whose payload type doesn't match the chosen format is skipped with an error log and the rest of the batch is still sent, in every mode. In `row_binary` mode there is an additional failure class: a JSON row whose values cannot be converted to the column types fails the whole batch (a half-written binary row would corrupt the rows after it). +Insert requests retry HTTP 408, 429, and 5xx responses, plus network and timeout errors. Other unsuccessful HTTP statuses fail immediately. `max_retries` is the total attempt limit, with at least one attempt even when set to `0`. Before retry number `n` (starting at 1), the delay is sampled from zero through `min(retry_delay * 2^n, 60)` seconds. With the defaults, there are at most three attempts and the first retry waits between zero and two seconds. The startup ping and, in `row_binary` mode, schema fetch use the same limit and backoff but retry every error. + +A message whose payload type does not match the chosen format is skipped with an error log; the rest of the batch is still sent. A batch with no serializable payloads returns success without an insert. In `row_binary` mode, a JSON row whose values cannot be converted to the column types fails the whole batch before any insert request, so it does not enter the plugin's insert retry loop. -Delivery is **at-least-once**: retries resend the full batch without an `insert_deduplication_token`, so a lost acknowledgement can insert the same rows twice. `ReplicatedMergeTree` tables suppress duplicates via implicit block-level deduplication in the common retry case. Plain `MergeTree` tables store them. If duplicates matter, deduplicate at the table engine or query level as described in the upstream [clickhouse_sink README](https://github.com/apache/iggy/tree/master/core/connectors/sinks/clickhouse_sink). +The runtime uses consumer auto-commit and does not replay a failed sink batch; see [sink guide](/docs/connectors/sinks/sink#sample-implementation). End-to-end at-least-once delivery is therefore not guaranteed. Plugin retries resend the same batch without an `insert_deduplication_token`, so a lost acknowledgement can also produce duplicate rows. +ClickHouse deduplication depends on the table engine, query settings, identical retry data, and the retained deduplication window. `ReplicatedMergeTree` enables a deduplication log by default. Non-replicated `MergeTree` can also deduplicate when `non_replicated_deduplication_window` is positive; its default is zero. See [ClickHouse insert deduplication](https://clickhouse.com/docs/concepts/features/operations/insert/deduplicating-inserts-on-retries) for the settings and limits. ## Transforms Transforms can be applied before inserting into ClickHouse. See the [transforms documentation](/docs/connectors/transforms) for the available types and their configuration. diff --git a/content/docs/connectors/sinks/delta.mdx b/content/docs/connectors/sinks/delta.mdx index 988c6d264f..0e46e79038 100644 --- a/content/docs/connectors/sinks/delta.mdx +++ b/content/docs/connectors/sinks/delta.mdx @@ -3,12 +3,18 @@ title: Delta Lake Sink description: "Write messages from Iggy streams into Delta Lake tables on local disk, S3, Azure Blob Storage or Google Cloud Storage." --- -The Delta Lake sink connector consumes messages from Iggy streams and writes them to Delta Lake tables on the local filesystem, AWS S3, Azure Blob Storage, or Google Cloud Storage. Each batch is flushed and committed as one atomic Delta transaction. +The Delta Lake sink connector consumes messages from Iggy streams and writes them to Delta Lake tables on the local filesystem, AWS S3, Azure Blob Storage, or Google Cloud Storage. Each successful nonempty batch is flushed and appended in one Delta transaction. This page is a curated subset of the documentation. The canonical reference is the upstream [delta_sink README](https://github.com/apache/iggy/tree/master/core/connectors/sinks/delta_sink) in the `apache/iggy` repository. ## Configuration +Use the broker credentials and runtime setup from the [sink guide](/docs/connectors/sinks/sink#configuration). Save the configuration below in the runtime's connector directory. From the matching 0.9.0/edge repository root, build the plugin: + +```bash +cargo build --release -p iggy_connector_delta_sink +``` + ```toml type = "sink" key = "delta" @@ -16,7 +22,6 @@ enabled = true version = 0 name = "Delta Lake sink" path = "target/release/libiggy_connector_delta_sink" -plugin_config_format = "toml" [[streams]] stream = "events" @@ -30,18 +35,28 @@ consumer_group = "delta_sink_connector" table_uri = "file:///tmp/iggy_delta_table" ``` -The target Delta table must already exist at `table_uri`: the connector opens it at startup and fails if it's missing. It doesn't create tables. Only JSON payloads are supported (`schema = "json"` on the stream). A batch containing another payload type fails. +Create the target table with your [Delta Lake tools](https://delta-io.github.io/delta-rs/usage/writing/) before starting the connector. The table must already exist at `table_uri`: the connector opens it at startup and fails if it's missing. It doesn't create tables. Only JSON payloads are supported (`schema = "json"` on the stream). A batch containing another payload type fails. + +For a table with columns `id` (long) and `name` (string), create the Iggy resources and send a row: + +```bash +./target/release/iggy --username iggy --password iggy stream create events +./target/release/iggy --username iggy --password iggy topic create events user_events 1 none 1d +./target/release/iggy --username iggy --password iggy message send --partition-id 0 events user_events '{"id":1,"name":"hello"}' +``` + +Use a `file:///...` URI for an absolute local path; bare filesystem paths are not accepted. ### Options | Option | Type | Default | Description | | ------ | ---- | ------- | ----------- | -| `table_uri` | string | **required** | Path or URI of the Delta table: `file://`, `s3://`, `az://`, or `gs://` | +| `table_uri` | string | **required** | Absolute URI of the Delta table: `file://`, `s3://`, `az://`, or `gs://` | | `storage_backend_type` | string | none | `s3`, `azure`, or `gcs`. Omit for local filesystem tables | -Each backend has its own credential options, required when `storage_backend_type` selects it: +When `storage_backend_type` selects a backend, the connector validates these credential options: -- **S3**: `aws_s3_access_key`, `aws_s3_secret_key`, `aws_s3_region`, `aws_s3_endpoint_url` (for S3-compatible stores like MinIO), `aws_s3_allow_http` (default `false`, for local development). +- **S3**: `aws_s3_access_key`, `aws_s3_secret_key`, and `aws_s3_region` are required. `aws_s3_endpoint_url` is optional, for S3-compatible stores like MinIO. `aws_s3_allow_http` is optional and defaults to `false`. - **Azure**: `azure_storage_account_name`, `azure_container_name`, and either `azure_storage_account_key` or `azure_storage_sas_token` (not both). - **GCS**: `gcs_service_account_key` (the service account JSON as a string, with the bucket taken from the `gs://` URI). @@ -60,15 +75,15 @@ aws_s3_region = "us-east-1" JSON values are coerced to the Delta table schema before writing: -- **Timestamp columns**: ISO 8601 / RFC 3339 strings (e.g. `"2021-11-11T22:11:58Z"`) are converted to microsecond timestamps. Numeric timestamps pass through. -- **String columns**: non-string values are converted to their string representation. -- Coercions apply recursively to nested structs and arrays. +- **Timestamp columns**: ISO 8601 / RFC 3339 strings (e.g. `"2021-11-11T22:11:58Z"`) are converted to microsecond timestamps. Integer timestamps pass through as epoch microseconds. Space-separated timestamps such as `"2021-11-11 22:11:58"` are interpreted as UTC. Invalid timestamp strings fail the batch. +- **String columns**: non-null, non-string values are converted to their JSON string representation. Nulls remain null. +- Coercions cover nested structs, arrays of strings or timestamps, and arrays of structs. Nested arrays, maps, and variant columns pass through without these coercions. The table schema is captured once at startup and not refreshed, so restart the connector after changing it. ## Batching & Transactions -Every poll batch is written to Parquet buffers and committed to the Delta log as a single transaction. The stream's `batch_length` therefore controls Delta commit granularity: small batches produce many small files and table versions, so prefer larger batches for production tables (and compact periodically). There is no retry on failed writes. A failed batch surfaces as a connector error. +Each nonempty batch passed to the plugin is written to Parquet buffers and appended in a single Delta log transaction. Empty batches create no transaction in `consume()`. The stream's `batch_length` is a polling limit, so it influences Delta commit granularity: small batches produce many small files and table versions, so prefer larger batches for production tables (and compact periodically). The plugin has no retry loop for failed batches, though the Delta library retries eligible commit conflicts and storage requests. A write or commit failure clears the writer buffers and returns a connector error. The runtime uses consumer auto-commit and does not replay that failed batch, so the Delta transaction does not provide an end-to-end at-least-once guarantee. ## Transforms diff --git a/content/docs/connectors/sinks/doris.mdx b/content/docs/connectors/sinks/doris.mdx index 7dc573914a..7272679f96 100644 --- a/content/docs/connectors/sinks/doris.mdx +++ b/content/docs/connectors/sinks/doris.mdx @@ -3,12 +3,18 @@ title: Apache Doris Sink description: "Load JSON messages from Iggy streams into a pre-created Apache Doris table through the Stream Load HTTP API." --- -The Apache Doris sink connector consumes JSON messages from Iggy streams and writes them to a pre-created Doris table via Doris's Stream Load HTTP API. Batches are loaded under deterministic labels so that in-request retries deduplicate instead of doubling rows. +The Apache Doris sink connector consumes JSON messages from Iggy streams and writes them to a pre-created Doris table via Doris's Stream Load HTTP API. Batches are loaded under deterministic labels so that retries deduplicate while Doris retains those labels. This page is a curated subset of the documentation. The canonical reference, including the label scheme, redirect security model, and operational guidance, is the upstream [doris_sink README](https://github.com/apache/iggy/tree/master/core/connectors/sinks/doris_sink) in the `apache/iggy` repository. ## Configuration +Use the broker credentials and runtime setup from the [sink guide](/docs/connectors/sinks/sink#configuration). Save the configuration below in the runtime's connector directory and replace the Doris password with your own. From the matching 0.9.0/edge repository root, build the plugin: + +```bash +cargo build --release -p iggy_connector_doris_sink +``` + ```toml type = "sink" key = "doris" @@ -16,7 +22,6 @@ enabled = true version = 0 name = "Doris sink" path = "target/release/libiggy_connector_doris_sink" -plugin_config_format = "toml" [[streams]] stream = "events" @@ -37,7 +42,15 @@ batch_size = 1000 timeout = "30s" ``` -The target database and table must be pre-created. The connector never issues DDL. `database` and `table` must match `[A-Za-z0-9_]+` or startup fails. Streams must use `schema = "json"`. +The target database and table must be pre-created. The connector never issues DDL. `database` and `table` must match `[A-Za-z0-9_]+` or startup fails. Streams must use `schema = "json"`. Startup validates the configuration but does not check the table or connection; these are first used when a batch arrives. + +For a table with columns `id` (`BIGINT`) and `name` (`STRING`), create the Iggy resources and send a row: + +```bash +./target/release/iggy --username iggy --password iggy stream create events +./target/release/iggy --username iggy --password iggy topic create events doris_events 1 none 1d +./target/release/iggy --username iggy --password iggy message send --partition-id 0 events doris_events '{"id":1,"name":"hello"}' +``` ### Common Options @@ -46,31 +59,37 @@ The target database and table must be pre-created. The connector never issues DD | `fe_url` | string | **required** | Doris frontend HTTP base URL, e.g. `http://localhost:8030` | | `database` | string | **required** | Target database, `[A-Za-z0-9_]+` | | `table` | string | **required** | Target table, `[A-Za-z0-9_]+` | -| `username` | string | **required** | Doris user with `LOAD_PRIV` on the table | +| `username` | string | **required** | Doris user authorized for Stream Load; check the grants required by your Doris version | | `password` | string | **required** | Doris user password, never logged | -| `batch_size` | u32 | `1000` | Maximum messages per Stream Load request | +| `batch_size` | u32 | `1000` | Maximum messages per Stream Load request; `0` is treated as `1` | | `output_format` | string | `json` | `json` or `csv`; CSV is opt-in for throughput and requires `columns` | | `columns` | string | unset | Forwarded as the `columns` Stream Load header; pins column order for CSV | -| `timeout` | string | `30s` | Per-request HTTP timeout | -| `max_retries` | u32 | `3` | Total Stream Load attempts per batch on transient failures | +| `timeout` | string | `30s` | Per-request HTTP timeout, including each redirect request | +| `max_retries` | u32 | `3` | Total Stream Load attempts per chunk on transient failures; `0` or `1` means one attempt | Further options cover the label prefix, connect timeout, retry backoff (`retry_delay`, `max_retry_delay`), `max_filter_ratio`, a `where` filter, and redirect security (`allow_insecure_redirect`, `allowed_redirect_hosts`). See the upstream [doris_sink README](https://github.com/apache/iggy/tree/master/core/connectors/sinks/doris_sink) for the full table with defaults. ## Output Formats - **`json`** (default): payloads are sent as a JSON array with `strip_outer_array: true`. Values map to columns by name, so the JSON shape must match the table (use `columns` if the JSON field names differ from the table columns or you need derived expressions). -- **`csv`**: opt-in for throughput. Doris CSV is positional, so `output_format = "csv"` **requires `columns`** to pin the column order. Startup fails without it. Rows are framed with control-character separators and quoted with `enclose`/`escape`, so embedded commas and newlines are safe. JSON `null` and missing keys become SQL `NULL`, and nested objects or arrays are stringified as JSON. +- **`csv`**: opt-in for throughput. Doris CSV is positional, so `output_format = "csv"` **requires `columns`** to pin the column order. Startup fails without it. List bare JSON field names in order before any derived expressions in `columns`; CSV reads only the leading names before the first `=` expression. Rows are framed with control-character separators and quoted with `enclose`/`escape`, so embedded commas and newlines are safe. JSON `null` and missing keys become SQL `NULL`, empty strings remain empty, and nested objects or arrays are stringified as JSON. The target Doris column types must accept the resulting values. ## Delivery Semantics -Each batch is loaded under a deterministic label derived from the stream, topic, partition, and offset range. Transient failures (HTTP 5xx/408/429, transport errors) are retried in-request with exponential backoff and jitter under the same label, which Doris deduplicates, so an ambiguous success is absorbed rather than doubled. Permanent errors (4xx, `Fail` status, malformed responses) are never retried. +Each poll is split into chunks of at most `batch_size` messages. Each chunk gets a deterministic label based on `label_prefix`, the target table, stream, topic, partition, and first/last offsets. Transient failures (HTTP 5xx/408/429, transport errors, empty or unreadable successful responses) are retried under the same label. Backoff starts at `retry_delay`, doubles for each retry, applies ±20% jitter, and never exceeds `max_retry_delay`. Other HTTP 4xx responses, `Fail` status, and nonempty malformed responses are permanent errors and are not retried. + +`Label Already Exists` is accepted only when the existing job is `FINISHED`; `RUNNING` and `CANCELLED` are retried. `Publish Timeout` is accepted as committed, although rows may not yet be visible. Deduplication lasts only while Doris retains the label. Check the retention settings for your Doris version; Doris 4.0.3 uses `streaming_label_keep_max_second` (default 12 hours) for Stream Load. See [Doris FE configuration](https://doris.apache.org/docs/4.x/admin-manual/config/fe-config/#streaming_label_keep_max_second). + +The runtime uses consumer auto-commit before the sink finishes. After a chunk fails, the plugin still attempts later chunks and returns the first error. The runtime logs and counts that error, adds no processed messages for the failed batch, and continues polling without replaying it. A crash or exhausted retry budget can therefore lose data. A non-JSON payload aborts the remaining chunks immediately. + +For a manual redrive, preserve the label inputs and chunk boundaries, including both `batch_length` and `batch_size`. Poll sizes can also vary, so unchanged settings alone do not guarantee the same labels. Changed boundaries or expired labels can produce duplicates. This is not an end-to-end exactly-once guarantee. -The runtime commits consumer offsets at poll time and doesn't inspect `consume()`'s return value, so delivery is **at-most-once** across polls: a failure that outlives the retry budget, or a crash mid-load, is not replayed. Keep `batch_size` stable across a manual redrive, since changing it shifts chunk boundaries and defeats label deduplication. +If `max_filter_ratio` permits malformed rows, Doris can accept a load while dropping those rows; the plugin logs a warning when Doris reports them. The `where` option deliberately excludes rows before loading. ## Security Notes - Use `https://` for `fe_url` in production: credentials travel as HTTP Basic auth. -- The connector preserves the `Authorization` header across the Doris FE-to-BE 307 redirect, validating the target first: scheme downgrades are refused unless `allow_insecure_redirect = true`, and `allowed_redirect_hosts` can pin redirects to known backend endpoints. +- The connector preserves the `Authorization` header across Doris FE-to-BE 307/308 redirects, validating each target first. When `fe_url` uses HTTPS, an HTTP redirect target is refused unless `allow_insecure_redirect = true`. `Location` must be absolute, and at most five redirects are followed per attempt. A nonempty `allowed_redirect_hosts` list restricts destinations to the listed hosts or `host:port` endpoints; an unset or empty list does not restrict hosts. - `columns` and `where` are forwarded verbatim to Doris and evaluated as SQL expressions. Keep this config trusted. ## Transforms diff --git a/content/docs/connectors/sinks/elasticsearch.mdx b/content/docs/connectors/sinks/elasticsearch.mdx index 0d99db5054..ca8831c66c 100644 --- a/content/docs/connectors/sinks/elasticsearch.mdx +++ b/content/docs/connectors/sinks/elasticsearch.mdx @@ -7,13 +7,19 @@ The Elasticsearch sink connector consumes messages from Iggy streams and indexes ## Configuration +Use the broker credentials and runtime setup from the [sink guide](/docs/connectors/sinks/sink#configuration). Save the configuration below in the runtime's connector directory and replace the Elasticsearch credentials with your own. From the matching 0.9.0/edge repository root, build the plugin: + +```bash +cargo build --release -p iggy_connector_elasticsearch_sink +``` + ```toml type = "sink" key = "elasticsearch-sink" enabled = true version = 1 name = "Elasticsearch Sink" -path = "/path/to/libiggy_connector_elasticsearch_sink.so" +path = "target/release/libiggy_connector_elasticsearch_sink" [[streams]] stream = "events" @@ -31,23 +37,35 @@ password = "changeme" create_index_if_not_exists = true ``` +Create the Iggy resources and send a JSON document: + +```bash +./target/release/iggy --username iggy --password iggy stream create events +./target/release/iggy --username iggy --password iggy topic create events logs 1 none 1d +./target/release/iggy --username iggy --password iggy message send --partition-id 0 events logs '{"timestamp":"2026-01-01T00:00:00Z","message":"hello","service_name":"example","level":"info"}' +``` + ### Plugin config options | Option | Type | Default | Description | | ------ | ---- | ------- | ----------- | | `url` | string | required | Elasticsearch cluster URL | | `index` | string | required | Target index name | -| `username` / `password` | string | none | Optional basic authentication credentials | +| `username` / `password` | string | none | Optional basic authentication; both values must be supplied to enable it | | `create_index_if_not_exists` | bool | `true` | Create the index on startup when it is missing | | `index_mapping` | table | none | Body of the create-index request (mappings, settings) | | `batch_size` | integer | unused | Accepted but currently not read by the connector | -| `timeout_seconds` | integer | unused | Accepted but currently not read by the connector | +| `timeout_seconds` | integer | `30` | HTTP request timeout in seconds, from connection through response body; `0` is treated as `1` | + +The stream-level `batch_length` governs batching: each consume cycle submits its supported messages in one bulk request. The connector performs no retries of its own. Request, HTTP-status, and response-decoding failures return an error; the runtime logs and counts it, adds no processed messages for that batch, and continues polling. Consumer auto-commit occurs before indexing completes, and the failed poll is not replayed. -The stream-level `batch_length` governs batching: each consume cycle bulk-indexes the batch of messages it receives. The connector performs no retries of its own. Failed batches are logged and counted. +Individual document errors inside a successful bulk response are logged and counted in the plugin's closing statistics, but the plugin still returns success. The runtime's processed-message count can therefore include documents rejected by Elasticsearch. Monitor the document-error logs as well as runtime errors. + +Bulk actions do not specify document IDs. Elasticsearch generates them, so replaying a message can create another document. The connector does not provide end-to-end exactly-once delivery. ### Index mapping -`index_mapping` is sent verbatim as the body of the create-index request, so it must be a structured object, not a JSON string embedded in TOML. Express it as nested TOML tables: +`index_mapping` is sent verbatim as the body of the create-index request, so it must be a structured object, not a JSON string embedded in TOML. It is used only when the connector creates an index; it does not update an existing index and is ignored when `create_index_if_not_exists = false`. That setting disables the startup existence check too. Express the mapping as nested TOML tables: ```toml [plugin_config.index_mapping.mappings.properties.timestamp] @@ -65,9 +83,11 @@ type = "keyword" ## Payload handling -- JSON payloads are indexed as documents directly. +- JSON objects are indexed as documents directly. Other JSON values are forwarded without being wrapped as objects and can be rejected by Elasticsearch. - Raw payloads are parsed as JSON when possible, and wrapped as `{ "data": "", "data_type": "raw" }` when that fails. - Text payloads are wrapped as `{ "text": "...", "data_type": "text" }`. - Other payload formats are skipped with a warning. -Each document is enriched with metadata fields before indexing: `_iggy_offset`, `_iggy_stream`, `_iggy_topic`, `_iggy_partition`, `_iggy_timestamp`, and `_iggy_headers` (when headers are present). +Object documents are enriched before indexing with `_iggy_offset`, `_iggy_stream`, `_iggy_topic`, `_iggy_partition`, and `_iggy_timestamp`. These fields replace matching payload fields. `_iggy_timestamp` is the connector's processing time in Unix epoch milliseconds, not the message's stored timestamp. `_iggy_headers` is added or replaced only when message headers are present. + +Use HTTPS when credentials must be protected in transit. The connector uses the Elasticsearch client's certificate verification; this configuration does not expose a switch to disable it. diff --git a/content/docs/connectors/sinks/http.mdx b/content/docs/connectors/sinks/http.mdx index 4220d6d5b8..55fd921a9b 100644 --- a/content/docs/connectors/sinks/http.mdx +++ b/content/docs/connectors/sinks/http.mdx @@ -9,7 +9,39 @@ This page is a curated subset of the documentation. The canonical reference, inc ## Configuration +Use the broker credentials and runtime setup from the [sink guide](/docs/connectors/sinks/sink). From the matching 0.9.0/edge repository root, build the plugin: + +```bash +cargo build --release -p iggy_connector_http_sink +``` + +For a local receiver, run this in a separate terminal: + +```bash +python3 - <<'PY_HTTP' +from http.server import BaseHTTPRequestHandler, HTTPServer + +class Receiver(BaseHTTPRequestHandler): + def do_POST(self): + body = self.rfile.read(int(self.headers["Content-Length"])) + print(body.decode("utf-8"), flush=True) + self.send_response(200) + self.end_headers() + +HTTPServer(("127.0.0.1", 9090), Receiver).serve_forever() +PY_HTTP +``` + +Save this configuration in the runtime's connector directory: + ```toml +type = "sink" +key = "http" +enabled = true +version = 0 +name = "HTTP sink" +path = "target/release/libiggy_connector_http_sink" + [[streams]] stream = "events" topics = ["notifications"] @@ -19,10 +51,18 @@ poll_interval = "100ms" consumer_group = "http_sink" [plugin_config] -url = "https://api.example.com/ingest" +url = "http://localhost:9090/ingest" batch_mode = "ndjson" ``` +Create the Iggy resources and send a JSON message, then start the connector runtime using the sink guide: + +```bash +./target/release/iggy --username iggy --password iggy stream create events +./target/release/iggy --username iggy --password iggy topic create events notifications 1 none 1d +./target/release/iggy --username iggy --password iggy message send --partition-id 0 events notifications '{"text":"hello"}' +``` + ### Common Options | Option | Type | Default | Description | @@ -36,13 +76,15 @@ batch_mode = "ndjson" Further options cover retries (`max_retries`, `retry_delay`, `retry_backoff_multiplier`, `max_retry_delay`), success status codes, payload size limits, TLS, connection pooling, health checks, and verbose logging. See the upstream [http_sink README](https://github.com/apache/iggy/tree/master/core/connectors/sinks/http_sink) for the full list with defaults. +The body size limit defaults to 10 MiB and applies after serialization: per message in `individual`/`raw`, or to the whole batch in `ndjson`/`json_array`. Setting `max_payload_size_bytes = 0` disables it. Oversized bodies fail without being sent. Optional startup health checks use the same URL, retry policy and success codes, with `HEAD` as the default method. + `GET` and `HEAD` are accepted but unusual for delivering data: the request still carries a body, which some servers reject. The sink logs a warning at startup when `GET` or `HEAD` is combined with a batch mode other than `individual`. ## Batch Modes The `batch_mode` option controls how messages from one poll cycle are delivered to the endpoint. -- **`individual`** (default): one HTTP request per message. Best for webhooks and endpoints that accept single events. With `batch_length = 50`, this produces 50 sequential round trips per poll cycle. +- **`individual`** (default): one HTTP request per message. Best for webhooks and endpoints that accept single events. With `batch_length = 50`, a full poll can produce 50 sequential requests, before retries; shorter polls and failures can produce fewer. - **`ndjson`**: all messages in one request, [newline-delimited JSON](https://github.com/ndjson/ndjson-spec). Best for bulk-ingestion endpoints. `Content-Type: application/x-ndjson`. - **`json_array`**: all messages as a single JSON array. Best for APIs that expect array payloads. `Content-Type: application/json`. - **`raw`**: raw bytes, one request per message. For non-JSON payloads (Protobuf, FlatBuffers, binary). The metadata envelope is not applied. `Content-Type: application/octet-stream`. @@ -63,14 +105,16 @@ When `include_metadata = true` (default), the JSON-mode payload is wrapped: "iggy_topic": "my_topic", "iggy_partition_id": 0 }, - "payload": { ... } + "payload": {"text": "hello"} } ``` - `iggy_id` is a 32-character lowercase hex string (no dashes). +- `iggy_timestamp` is the message timestamp in Unix epoch microseconds. `include_checksum` and `include_origin_timestamp` add the corresponding metadata fields; both default to `false`. +- Nonempty message headers become `iggy_headers`: non-raw values are strings, and raw values use a base64 `data` object with `iggy_header_encoding = "base64"`. - Set `include_metadata = false` to send the payload without wrapping (useful when the downstream service expects bare JSON, e.g. Slack webhooks). -Non-JSON payloads (`raw`, `flatbuffer`, `proto`, and `avro` schemas) cannot be embedded in a JSON body directly, so they are base64-encoded with an explicit marker: +In JSON batch modes, raw, FlatBuffer, Protobuf and Avro payload variants are base64-encoded with an explicit marker: ```json { @@ -79,35 +123,41 @@ Non-JSON payloads (`raw`, `flatbuffer`, `proto`, and `avro` schemas) cannot be e } ``` -With the envelope enabled, this object becomes the `payload` field shown above. +With the envelope enabled, this object becomes the `payload` field shown above. JSON values are reserialized and text payloads become JSON strings. + +The runtime applies stream decoding and configured transforms before the sink formats its requests. To forward opaque message bytes, use `schema = "raw"` with `batch_mode = "raw"` and no transforms. See the [SDK format limits](/docs/connectors/sdk) when selecting other schemas. -The connector doesn't require any particular message structure on input. The envelope is applied on the way out, not expected on the way in, so your producers can publish whatever they like. +The connector doesn't expect the metadata envelope on input; it adds it on the way out. Producers must supply payloads accepted by the configured stream decoder and the receiving endpoint. ## Authentication -The HTTP sink supports authentication via custom headers under `[plugin_config.headers]`. All headers are sent with every request, including health checks. +The HTTP sink supports authentication via custom headers under `[plugin_config.headers]`. Custom headers are sent with data requests and health checks, except `Content-Type`: the sink ignores a configured value and sets it from the batch mode for data requests. ```toml [plugin_config.headers] Authorization = "Bearer eyJhbGciOiJSUzI1NiIs..." ``` -Any header works the same way: use `x-api-key = "my-secret-api-key"` for API-key schemes, or `Authorization = "Basic dXNlcjpwYXNzd29yZA=="` (base64 of `username:password`) for basic auth. Multiple headers are combined per request. For secrets, prefer environment variable overrides at the process level (see the upstream README) to keep tokens out of `config.toml`. +Other valid headers work the same way: use `x-api-key = "my-secret-api-key"` for API-key schemes, or `Authorization = "Basic dXNlcm5hbWU6cGFzc3dvcmQ="` (base64 of `username:password`) for basic auth. Multiple headers are combined per request. Configure header secrets in a protected connector TOML file. The local provider supports environment overrides for flat plugin fields, such as `URL`, but not nested headers or JSON objects. `HEADERS_AUTHORIZATION` creates an unused flat field, and a JSON object supplied through `HEADERS` is parsed as a string and fails plugin initialization. ## Retry & Delivery Semantics -Failed requests are retried with exponential backoff: `retry_delay`, multiplied by `retry_backoff_multiplier` per attempt, capped at `max_retry_delay`, for up to `max_retries` attempts. +Transient failures allow up to `max_retries` additional attempts after the initial request: the default `3` means at most four attempts. Retry delays use full jitter from zero up to `min(retry_delay * retry_backoff_multiplier^n, max_retry_delay)`, where `n` starts at zero for the first retry. With the defaults, the three delay caps are 1, 2 and 4 seconds. - **Transient errors** (retried): network errors, HTTP 429, 500, 502, 503, 504. - **Non-transient errors** (fail immediately): HTTP 400, 401, 403, 404, 405, etc. - **`success_status_codes`** (default `[200, 201, 202, 204]`) short-circuits retries: any status code in this set is treated as success and is never retried, even codes that are normally transient. Placing `429` in the set makes the sink accept rate-limited responses as delivered. The sink warns at startup about such overlaps. -- **HTTP 429 `Retry-After`**: the header is logged but not honored. Retry timing always uses the computed backoff. +- **HTTP 429 `Retry-After`**: on unsuccessful responses, the header is logged but not honored. Retry timing always uses the computed backoff. - **Partial delivery** (`individual`/`raw` modes): after a fixed number of consecutive HTTP failures (3 at the time of writing, not configurable), the remainder of the batch is aborted to avoid hammering a dead endpoint. -The connector runtime commits consumer-group offsets when messages are polled, before `consume()` runs, and doesn't inspect its return value, so the effective delivery guarantee is **at-most-once** at the runtime level. The sink's internal retry loop provides best-effort delivery within each `consume()` call. +The connector runtime uses consumer auto-commit before `consume()` completes. A plugin error is logged and counted, the failed batch adds no processed messages, and polling continues without replaying that batch. Earlier messages or requests from the same batch may already have succeeded. A crash or exhausted retry budget can lose messages; a retry after an ambiguous response can deliver them again. There is no end-to-end at-least-once or exactly-once guarantee. + +HTTP success is determined by the response status. The sink does not inspect a successful response body for per-item failures. ## Example Configurations +Replace the corresponding blocks in the complete configuration above, and supply your own endpoint URLs and credentials. + ### Webhook (Slack) ```toml @@ -118,7 +168,7 @@ include_metadata = false max_retries = 5 ``` -The sink performs no outbound payload transformation. `include_metadata = false` only skips the envelope. The message payload is delivered exactly as produced, so your producer must publish JSON already in the shape Slack expects (for example `{"text": "..."}`). +Use `schema = "json"` and publish JSON in the shape [Slack expects](https://docs.slack.dev/messaging/sending-messages-using-incoming-webhooks/), for example `{"text":"hello"}`. `include_metadata = false` skips the envelope, but JSON is still reserialized; the original bytes and whitespace are not preserved. ### Bulk REST API Ingestion @@ -156,7 +206,7 @@ timeout = "5s" One `iggy-connectors` runtime process can host many connectors: the local config provider imports every connector TOML file under its `config_dir`, and each entry gets its own plugin instance and consume tasks in the same process. Each connector entry has exactly one `[plugin_config]` block and therefore one destination URL. To send to multiple destinations, add multiple HTTP sink entries with distinct keys and URLs (in the same runtime or separate ones). For fan-out from a single topic to multiple endpoints, point multiple sink entries at the same topic with different `consumer_group` names so each maintains its own offset. -Throughput is dominated by batch-mode choice: the runtime calls `consume()` sequentially within each topic task, so `individual` and `raw` modes perform N round trips per poll cycle while `ndjson`/`json_array` collapse the same batch into one request. Connection pooling and HTTP keep-alive are enabled by default. Tune `max_connections` and `batch_length` together for high-throughput targets. +Throughput is dominated by batch-mode choice: the runtime calls `consume()` sequentially within each topic task, so a full N-message batch uses N requests in `individual`/`raw`, or one in `ndjson`/`json_array`, before retries or skipped messages. Connection pooling and HTTP keep-alive are enabled by default. `max_connections` limits idle connections retained per host, not concurrent requests. Topic tasks can issue requests concurrently; each topic awaits its own batch before continuing. Tune pooling and `batch_length` for the receiving service. For multi-instance deployment patterns (Docker, Kubernetes, fan-out topologies), connection pool tuning, and full performance analysis, see the upstream [http_sink README](https://github.com/apache/iggy/tree/master/core/connectors/sinks/http_sink). @@ -168,6 +218,6 @@ For multi-instance deployment patterns (Docker, Kubernetes, fan-out topologies), - **No per-topic URL routing**: all topics in one connector instance share the same `url`. For routing, deploy multiple instances. - **No OAuth2 / OIDC token refresh**: bearer tokens are static. Use an auth proxy for services that require token rotation. - **No mTLS client certificates**: terminate mTLS at a sidecar proxy for production use. -- **Plaintext secrets in config**: header values are stored verbatim in `config.toml`. Use environment variable overrides at the process level for sensitive values. +- **Plaintext secrets in config**: header values are stored verbatim in `config.toml`. Mount or generate a protected connector TOML file containing the required header values before starting the runtime. For the complete list and implementation context, see the upstream [http_sink README](https://github.com/apache/iggy/tree/master/core/connectors/sinks/http_sink#known-limitations). diff --git a/content/docs/connectors/sinks/iceberg.mdx b/content/docs/connectors/sinks/iceberg.mdx index d0b74a7bfa..6b3c33f51d 100644 --- a/content/docs/connectors/sinks/iceberg.mdx +++ b/content/docs/connectors/sinks/iceberg.mdx @@ -15,50 +15,108 @@ The Iceberg Sink Connector allows you to consume messages from Iggy topics and s ## Configuration example +Use the broker credentials and runtime setup from the [sink guide](/docs/connectors/sinks/sink#configuration). From the matching 0.9.0/edge repository root, build the plugin: + +```bash +cargo build --release -p iggy_connector_iceberg_sink +``` + +The example requires a REST catalog at `http://localhost:8181` and S3-compatible storage at `http://localhost:9000`, configured with the credentials below. Create the storage bucket, namespace and target table before starting the connector. The sink does not create them. + +For a local catalog without authentication, create `nyc.users` with `id` (long) and `name` (string): + +```bash +curl --fail-with-body --request POST http://localhost:8181/v1/namespaces \ + --header 'Content-Type: application/json' \ + --data '{"namespace":["nyc"]}' +curl --fail-with-body --request POST http://localhost:8181/v1/namespaces/nyc/tables \ + --header 'Content-Type: application/json' \ + --data '{"name":"users","schema":{"type":"struct","fields":[{"id":1,"name":"id","type":"long","required":true},{"id":2,"name":"name","type":"string","required":true}]}}' +``` + +Save this complete connector file in the runtime's connector directory: + ```toml +type = "sink" +key = "iceberg" +enabled = true +version = 0 +name = "Iceberg sink" +path = "target/release/libiggy_connector_iceberg_sink" + +[[streams]] +stream = "events" +topics = ["users"] +schema = "json" +batch_length = 100 +poll_interval = "5ms" +consumer_group = "iceberg_sink_connector" + [plugin_config] tables = ["nyc.users"] catalog_type = "rest" warehouse = "warehouse" uri = "http://localhost:8181" -dynamic_routing = true +dynamic_routing = false dynamic_route_field = "db_table" store_url = "http://localhost:9000" store_access_key_id = "admin" store_secret_access_key = "password" store_region = "us-east-1" store_class = "s3" +store_path_style_access = true +``` + +Create the Iggy resources and send a row: + +```bash +./target/release/iggy --username iggy --password iggy stream create events +./target/release/iggy --username iggy --password iggy topic create events users 1 none 1d +./target/release/iggy --username iggy --password iggy message send --partition-id 0 events users '{"id":1,"name":"hello"}' ``` ## Configuration Options -- **tables**: The names of the Iceberg tables you want to statically route Iggy messages to. The name should include the table’s namespace, separated by a dot (`.`). +All options below are required except the credential pair and `store_path_style_access`. `tables` and `dynamic_route_field` must be present even when the selected routing mode does not use them. + +- **tables**: The names of the Iceberg tables you want to statically route Iggy messages to. The name should include the table’s namespace, separated by a dot (`.`). Ignored when `dynamic_routing = true`. - **catalog_type**: The type of catalog you are routing data to. **Currently, only REST catalogs are fully supported.** -- **warehouse**: The name of the bucket or warehouse where Iggy will upload data files. +- **warehouse**: The warehouse value sent to the REST catalog. Its meaning depends on the catalog service; data file destinations come from each table’s metadata. - **uri**: The URI of the Iceberg catalog. - **dynamic_routing**: Enables dynamic routing. See more details later in this document. -- **dynamic_route_field**: The name of the message field that specifies the Iceberg table to route data to. See more details below. +- **dynamic_route_field**: The top-level message field that specifies the Iceberg table to route data to. Ignored in static mode. See more details below. - **store_url**: The URL of the object storage for data uploads. -- **store_access_key_id**: The access key ID of the object storage. -- **store_secret_access_key**: The secret key used to upload data to the object storage. +- **store_access_key_id**: The optional access key ID of the object storage. +- **store_secret_access_key**: The optional secret key used to upload data to the object storage. Supply both credential fields or omit both to use the default AWS credential provider chain. - **store_region**: The region of the object storage. Required. For S3-compatible stores that ignore it, supply any placeholder value. - **store_class**: The storage class to use. **Currently, only S3-compatible storage is supported.** +- **store_path_style_access**: Use path-style S3 URLs (`http://host/bucket/key`). Defaults to `true`; set to `false` for stores that require virtual-hosted-style URLs. + +## Static Routing + +With `dynamic_routing = false`, every batch is copied to every successfully loaded table in `tables`. Invalid names and tables that cannot be loaded are skipped at startup. Startup fails if no table can be loaded. The writer uses each table's schema and default partition spec captured at startup, so restart the connector after changing them. + ## Dynamic Routing If you don't know the names of the Iceberg tables you want to route data to in advance, you can use the dynamic routing feature. -Insert a field in your Iggy messages with the name of the Iceberg table the message should be routed to. The Iggy connector will parse this field at runtime and route the message to the correct table. +Insert a top-level field in your JSON messages with the name of the Iceberg table the message should be routed to. The Iggy connector will parse this field at runtime and route the message to the correct table. The Iggy Iceberg Connector will skip messages in the following cases: -- The table declared in the message field does not exist. -- The message does not contain the field specified in the `dynamic_route_field` configuration option. +- The table declared in the message field cannot be loaded, including missing tables and catalog lookup failures. +- The table name has no namespace or contains an empty name component. +- The message does not contain the field specified in the `dynamic_route_field` configuration option, or is not a JSON object. + +A lookup failure skips the affected message; it does not fail the batch. Each batch loads its destination tables again. The route field is included in the JSON row, but is ignored by the writer if the target schema has no matching column. ### Dynamic routing configuration example +Replace `[plugin_config]` in the complete connector file above and append the transform below. `tables = []` is intentional: the route field chooses the destination in this mode. + ```toml [plugin_config] -tables = [""] +tables = [] catalog_type = "rest" warehouse = "warehouse" uri = "http://localhost:8181" @@ -69,6 +127,7 @@ store_access_key_id = "admin" store_secret_access_key = "password" store_region = "us-east-1" store_class = "s3" +store_path_style_access = true [transforms.add_fields] enabled = true @@ -86,9 +145,17 @@ Example: - Namespace: `nyc` - Table name: `users` +## Batches and table commits + +Rows are converted to Arrow using the target table schema, written to Parquet, then appended in an Iceberg transaction per destination table. Partitioned tables split a batch by the default partition spec. Writers can roll over into multiple files, so there is no fixed one-file-per-batch guarantee for either partitioned or unpartitioned tables. + +Fan-out is not atomic across tables. A write or commit error stops work on the remaining tables in that batch; an earlier table may already be committed. The Iceberg library refreshes metadata before committing and retries eligible commit errors according to the table's retry properties. The plugin does not replay a failed batch. The runtime records a plugin error and continues polling with consumer auto-commit, so messages can be lost after a failed or skipped write. Replaying an already committed batch can create duplicates. + ## Source Compatibility -The Iceberg sink expects **flat JSON** where each top-level key maps directly to a column in the target Iceberg table schema. Sources that wrap row data in an envelope (with metadata fields alongside a nested data object) are not directly compatible: the Arrow JSON reader will map envelope keys to table columns, producing nulls or schema errors. +Use `schema = "json"` on the stream. Each JSON object represents a table row; nested objects and arrays are supported when they match the table schema. Unknown fields are ignored. Missing nullable fields become null; missing required fields or incompatible values fail the table write. + +Sources that wrap row data in an envelope need a transform when the table schema describes the inner row. Otherwise, the Arrow JSON reader maps envelope keys to table columns, producing nulls or schema errors. If your source emits envelope-wrapped JSON, use the `unwrap_envelope` transform to extract the inner data field before it reaches the sink: diff --git a/content/docs/connectors/sinks/influxdb.mdx b/content/docs/connectors/sinks/influxdb.mdx index c5848e1003..f76c9d4667 100644 --- a/content/docs/connectors/sinks/influxdb.mdx +++ b/content/docs/connectors/sinks/influxdb.mdx @@ -9,6 +9,14 @@ This page is a curated subset of the documentation. The canonical reference is t ## Configuration +Use the broker credentials and runtime setup from the [sink guide](/docs/connectors/sinks/sink#configuration). From the matching 0.9.0/edge repository root, build the plugin: + +```bash +cargo build --release -p iggy_connector_influxdb_sink +``` + +Create the InfluxDB organization and bucket (V2), or database (V3), and provide a token with write access. Save the following connector file in the runtime's connector directory, replacing the URL and token for your deployment. Startup checks `GET /health` with the configured token, using `Token` authentication for V2 and `Bearer` for V3. A successful health check does not verify write permissions. + ```toml type = "sink" key = "influxdb" @@ -37,8 +45,18 @@ precision = "us" batch_size = 500 ``` +Create the Iggy resources and send a row: + +```bash +./target/release/iggy --username iggy --password iggy stream create events +./target/release/iggy --username iggy --password iggy topic create events influx_events 1 none 1d +./target/release/iggy --username iggy --password iggy message send --partition-id 0 events influx_events '{"sensor":"temperature","value":21.5}' +``` + InfluxDB V2 organizes data as `org` + `bucket`. V3 uses a single `db` field instead. Omitting `version` defaults to `"v2"` for backward compatibility. The write body, batching, and retry behavior are identical between versions. The connector handles the differing endpoints, auth header styles, and precision spellings internally. +For V3, replace the entire `[plugin_config]` section. Unknown options are rejected, so remove the V2 `org` and `bucket` fields: + ```toml # V3 variant: org/bucket are replaced by db [plugin_config] @@ -57,26 +75,46 @@ token = "replace_with_secret_token" | `org` | string | **required** (v2) | Organization name | | `bucket` | string | **required** (v2) | Target bucket | | `db` | string | **required** (v3) | Target database | -| `token` | string | **required** | API token, never logged | +| `token` | string | **required** | API token; redacted in the plugin’s debug representation | | `measurement` | string | `iggy_messages` | Line-protocol measurement name | | `precision` | string | `us` | Timestamp precision: `ns`, `us`, `ms`, or `s` | -| `batch_size` | u32 | `500` | Messages per write request | +| `batch_size` | u32 | `500` | Maximum messages per write request; `0` behaves as `1` | | `payload_format` | string | `json` | `json`, `text`, or `base64` | -| `include_metadata` | bool | `true` | Inject stream/topic/partition fields into each point (the `offset` tag is always written regardless) | +| `include_metadata` | bool | `true` | Include stream/topic/partition metadata as tags or fields; independent of checksum and origin timestamp | + +The `include_checksum`, `include_origin_timestamp`, `include_stream_tag`, `include_topic_tag`, and `include_partition_tag` flags also default to `true`. -The `include_checksum`, `include_origin_timestamp`, `include_stream_tag`, `include_topic_tag`, and `include_partition_tag` flags also default to `true`. Disable them individually to slim down points. Resilience options cover `timeout` (`30s`), `max_retries` (`3`), `retry_delay` / `retry_max_delay`, startup health-check retries, and a circuit breaker (`circuit_breaker_threshold` `5`, `circuit_breaker_cool_down` `30s`). See the upstream [influxdb_sink README](https://github.com/apache/iggy/tree/master/core/connectors/sinks/influxdb_sink) for the full list. +- `message_id` is always a string field and `offset` is always a tag. +- With `include_metadata = true`, stream, topic and partition are tags by default. Disabling an individual `include_*_tag` flag writes that value as an `iggy_stream`, `iggy_topic`, or `iggy_partition` field instead. +- `include_metadata = false` omits those three values. It does not disable the independently controlled `iggy_checksum` and `iggy_origin_timestamp` fields. + +Point timestamps come from the Iggy message timestamp in microseconds, converted to `precision`. A zero timestamp uses the current wall-clock time. Millisecond and second precision discard finer timestamp digits. + +InfluxDB identifies points by measurement/table, tags and timestamp. Removing stream, topic or partition tags can merge distinct messages that share an offset and timestamp; moving those values to fields does not preserve their identity. See the [V2](https://docs.influxdata.com/influxdb/v2/reference/syntax/line-protocol/#duplicate-points) and [V3](https://docs.influxdata.com/influxdb3/core/reference/line-protocol/#duplicate-points) duplicate-point rules. + +Resilience options include `timeout` (`30s`), `max_retries` (`3` total write attempts), `retry_delay` (`1s`), `retry_max_delay` (`5s`), `max_open_retries` (`10` total health-check attempts), `open_retry_max_delay` (`60s`), `circuit_breaker_threshold` (`5`), and `circuit_breaker_cool_down` (`30s`). Invalid duration strings warn and fall back to `1s`. See the upstream [influxdb_sink README](https://github.com/apache/iggy/tree/master/core/connectors/sinks/influxdb_sink) for the full list. ## Payload Formats - **`json`** (default): the message payload is validated as JSON, compact-serialized, and written as a single `payload_json` string field (individual JSON fields do not become separate field entries). -- **`text`**: the payload is written as a single `payload_text` string field. -- **`base64`**: the raw payload bytes are base64-encoded into a `payload_base64` string field. +- **`text`**: the payload must be valid UTF-8 and is written as a single `payload_text` string field. +- **`base64`**: the payload bytes are base64-encoded into a `payload_base64` string field. + +`payload_format` selects the stored representation after stream decoding and transforms. For arbitrary bytes, use `schema = "raw"` with `payload_format = "base64"` and no payload-changing transform. For plain text, use `schema = "text"` with `payload_format = "text"`. JSON stream decoding can change the original byte representation. -An unrecognized `payload_format` logs a warning and falls back to `json`. +Text carriage returns and newlines become the literal sequences `\r` and `\n` in the stored string. Use base64 if those original bytes must round-trip. Tabs in measurement names or tag values are rejected; quoted text fields allow tabs. + +Format names are case-insensitive; `utf8` aliases `text` and `raw` aliases `base64`. An unrecognized `payload_format` logs a warning and falls back to `json`. ## Reliability -Messages are serialized to line protocol and each polled batch is split into chunks of at most `batch_size` messages, one HTTP POST per chunk. Transient errors (429 and 5xx) are retried with exponential backoff, and after `circuit_breaker_threshold` consecutive failures the connector stops issuing writes until the cool-down window elapses, then probes again. The startup health check retries with its own backoff, so the connector tolerates InfluxDB starting after it. +Messages are serialized to line protocol and each polled batch is split into chunks of at most `batch_size` messages, one HTTP POST per chunk. The final partial chunk is written immediately; the connector does not accumulate messages across polls. A serialization error rejects its entire chunk. Later chunks are still attempted after a failure, and the first error is returned after the loop. + +HTTP 429, all 5xx responses, and network errors are retried. `max_retries = 3` allows the initial attempt plus two retries; `0` and `1` both allow one attempt. Backoff starts at `retry_delay`, doubles, adds ±20% jitter, and is capped by `retry_max_delay`. An integer-seconds `Retry-After` on a 429 overrides that cap; HTTP-date values are ignored. Startup health checks use a separate attempt budget and cap, retrying any failed check. + +The circuit breaker counts at most one failure per consumed batch, based on its first error. A permanent HTTP error does not increment the counter; a fully successful batch resets it. While the breaker is open, incoming batches fail without a write. After the cool-down window, writes resume and the failure counter resets. + +The runtime records a plugin error and continues polling with consumer auto-commit. Failed chunks and batches skipped by the circuit breaker are not queued for replay, so retries and the circuit breaker do not provide an end-to-end at-least-once guarantee. Other chunks from the same batch may already have reached InfluxDB. ## Transforms diff --git a/content/docs/connectors/sinks/meilisearch.mdx b/content/docs/connectors/sinks/meilisearch.mdx index 26b996b275..eae9e6fa1e 100644 --- a/content/docs/connectors/sinks/meilisearch.mdx +++ b/content/docs/connectors/sinks/meilisearch.mdx @@ -9,6 +9,14 @@ This page is a curated subset of the documentation. The canonical reference is t ## Configuration +Use the broker credentials and runtime setup from the [sink guide](/docs/connectors/sinks/sink#configuration). From the matching 0.9.0/edge repository root, build the plugin: + +```bash +cargo build --release -p iggy_connector_meilisearch_sink +``` + +Save this connector file in the runtime's connector directory. Replace the URL and API key with your Meilisearch endpoint and credentials. The key needs permission to inspect the index, submit documents and read tasks, plus create the index when needed. Startup checks health, checks the index, and waits for index creation to succeed even when `wait_for_tasks = false`. Set `primary_key` to match an existing index. A mismatch only produces a startup warning; the connector does not change the index key, and document tasks can fail. + ```toml type = "sink" key = "meilisearch" @@ -27,7 +35,7 @@ poll_interval = "5ms" consumer_group = "meilisearch_sink" [plugin_config] -url = "https://meilisearch.example.com" +url = "http://localhost:7700" index = "iggy_messages" api_key = "replace_with_secret_key" primary_key = "iggy_id" @@ -35,34 +43,48 @@ document_action = "replace" batch_size = 1000 ``` +Create the Iggy resources and send a document: + +```bash +./target/release/iggy --username iggy --password iggy stream create events +./target/release/iggy --username iggy --password iggy topic create events search_events 1 none 1d +./target/release/iggy --username iggy --password iggy message send --partition-id 0 events search_events '{"title":"First event","category":"example"}' +``` + ### Common Options | Option | Type | Default | Description | | ------ | ---- | ------- | ----------- | -| `url` | string | **required** | Meilisearch base URL; paths and query strings are ignored | +| `url` | string | **required** | Meilisearch base URL; paths, query strings and fragments are ignored; missing scheme defaults to HTTP | | `index` | string | **required** | Target index UID | | `api_key` | string | none | API key sent as `Authorization: Bearer`; use HTTPS for non-local hosts | | `primary_key` | string | `iggy_id` | Index primary key field | | `document_action` | string | `replace` | `replace` (add-or-replace) or `update` (add-or-update) | | `create_index_if_not_exists` | bool | `true` | Create the index during startup when missing | | `include_metadata` | bool | `true` | Add reserved `iggy_*` provenance fields to each document | -| `batch_size` | usize | `1000` | Maximum documents per Meilisearch request | -| `wait_for_tasks` | bool | `true` | Poll Meilisearch tasks to a terminal state before returning | -| `timeout` | string | `30s` | Request timeout | +| `batch_size` | usize | `1000` | Maximum documents per Meilisearch request; `0` behaves as `1` | +| `wait_for_tasks` | bool | `true` | Wait for indexing success or failure, bounded by `task_timeout` | +| `timeout` | string | `30s` | Total deadline per retried SDK operation; per attempt for health checks | -Further options cover task polling (`task_timeout` `30s`, `task_poll_interval` `100ms`) and transient retries (`max_retries` `3`, `retry_delay` `500ms`, `max_retry_delay` `5s`, `max_open_retries` `5`). See the upstream [meilisearch_sink README](https://github.com/apache/iggy/tree/master/core/connectors/sinks/meilisearch_sink) for details. +Further options cover task polling (`task_timeout` `30s`, `task_poll_interval` `100ms`) and transient retries (`max_retries` `3`, `retry_delay` `500ms`, `max_retry_delay` `5s`, `max_open_retries` `5`). Both retry limits count retries after the first request, so `max_retries = 3` permits up to four attempts within the operation deadline. Backoff doubles from `retry_delay`, adds ±20% jitter and is capped at `max_retry_delay`; reversed delay bounds are swapped. Invalid duration strings warn and fall back to `1s`. There is no single deadline for the whole startup sequence. See the upstream [meilisearch_sink README](https://github.com/apache/iggy/tree/master/core/connectors/sinks/meilisearch_sink) for details. ## Document Mapping -JSON object payloads are indexed as documents directly. JSON arrays and scalars are wrapped in a `value` field, since Meilisearch documents must be objects. Raw payloads are parsed as JSON when possible and otherwise indexed as base64 data. Text payloads land in a `text` field. Records with unsupported payload schemas are skipped with a warning and counted as errors. +JSON object payloads are indexed as documents directly. JSON arrays and scalars are wrapped in a `value` field, since Meilisearch documents must be objects. Raw payloads are parsed as JSON when possible and otherwise indexed in `data` as base64, with `data_type = "raw"` and `data_encoding = "base64"`. Text payloads land in `text` with `data_type = "text"`. These mappings apply after stream decoding and transforms; use `schema = "raw"` or `schema = "text"` for those representations. Unsupported payload variants are skipped with a warning and counted in the plugin's private error counter. A successful callback after these drops can still count the whole batch as processed in runtime statistics. + +When the configured primary key is absent from a document, the connector injects a stable value derived from the stream, topic, partition, offset, and message ID, avoiding Meilisearch primary-key inference failures. An existing key is preserved, including an invalid or null value that Meilisearch may reject, unless a reserved metadata field below overwrites it. Use unique values that meet [Meilisearch's document ID rules](https://www.meilisearch.com/docs/resources/internals/primary_key). Reusing a key with `replace` replaces the whole document, removing omitted fields; `update` keeps fields omitted from the new document. Both actions can insert a new document. -When the configured primary key is absent from a document, the connector injects a stable value derived from the stream, topic, partition, offset, and message ID, avoiding Meilisearch primary-key inference failures. If your payloads carry their own primary key, its values must be unique. Otherwise add-or-replace semantics collapse distinct messages into one document. +With `include_metadata = true`, stream, topic, partition, offset, checksum, message ID, timestamps and available headers are written as reserved `iggy_*` fields after payload parsing, overwriting same-named payload fields. The exception is a supplied `iggy_id` when it is the configured primary key. With a different primary key, `iggy_id` holds the generated Iggy identity. Avoid other reserved metadata fields as your primary key because metadata overwrites them. -With `include_metadata = true`, reserved `iggy_*` fields (stream, topic, partition, offset, checksum, timestamps) are written after payload parsing and overwrite same-named payload fields. +`iggy_checksum` and `iggy_message_id` are strings; offsets and message timestamps remain numbers. Message timestamps are microseconds, while `iggy_ingested_at` uses current wall-clock milliseconds. Disabling metadata leaves payload fields intact and still injects a missing primary key. ## Delivery Semantics -The runtime commits consumer offsets when messages are polled and doesn't gate commits on `consume()`'s result, so the effective guarantee is **at-most-once** on sink errors. The retry settings provide best-effort retries within a single batch. Setting `wait_for_tasks = false` makes indexing fire-and-forget: submission succeeds before Meilisearch confirms indexing, and later task failures are neither observed nor retried. +Each polled batch is split into requests of at most `batch_size` documents, with no accumulation across polls. The first failed chunk stops the loop; later chunks are not attempted. Earlier chunks may already be indexed. + +Transient submission and task-status errors are retried within their budgets. A failed indexing task is returned as an error without resubmitting it. A task timeout does not cancel the remote task, so it may finish later. The runtime records plugin errors and continues polling with consumer auto-commit; it does not replay failed batches. This does not provide an end-to-end at-least-once guarantee, and retries or manual replay after uncertain outcomes can repeat writes. + +Setting `wait_for_tasks = false` returns after submission, before Meilisearch confirms indexing. Later task failures are neither observed nor retried by the connector. Index-creation tasks are still awaited during startup. Closing the connector does not wait for outstanding document tasks. ## Transforms diff --git a/content/docs/connectors/sinks/mongodb.mdx b/content/docs/connectors/sinks/mongodb.mdx index c7e75f60fc..ed15b21eea 100644 --- a/content/docs/connectors/sinks/mongodb.mdx +++ b/content/docs/connectors/sinks/mongodb.mdx @@ -7,13 +7,21 @@ The MongoDB sink connector writes messages from Iggy streams to a MongoDB databa ## Configuration +Use the broker credentials and runtime setup from the [sink guide](/docs/connectors/sinks/sink#configuration). From the matching 0.9.0/edge repository root, build the plugin: + +```bash +cargo build --release -p iggy_connector_mongodb_sink +``` + +Save this connector file in the runtime's connector directory and replace the MongoDB URI and destination names for your deployment. Startup parses the URI and pings the target database. `max_pool_size`, when set, overrides the URI's pool setting; other connection options come from the MongoDB driver and URI. + ```toml type = "sink" key = "mongodb-sink" enabled = true version = 1 name = "MongoDB Sink" -path = "/path/to/libiggy_connector_mongodb_sink.so" +path = "target/release/libiggy_connector_mongodb_sink" verbose = false [[streams]] @@ -39,6 +47,14 @@ max_retries = 3 retry_delay = "1s" ``` +Create the Iggy resources and send a message: + +```bash +./target/release/iggy --username iggy --password iggy stream create my-stream +./target/release/iggy --username iggy --password iggy topic create my-stream my-topic 1 none 1d +./target/release/iggy --username iggy --password iggy message send --partition-id 0 my-stream my-topic '{"hello":"mongodb","internal_field":"remove me"}' +``` + ## Plugin config options | Option | Type | Default | Description | @@ -47,15 +63,37 @@ retry_delay = "1s" | `database` | string | required | Target database name | | `collection` | string | required | Target collection name | | `max_pool_size` | u32 | driver default | Maximum connections in the MongoDB client pool | -| `batch_size` | u32 | `100` | Number of documents to insert per batch | -| `include_metadata` | bool | `true` | Include message metadata (offset, timestamp) in documents | +| `batch_size` | u32 | `100` | Maximum documents per insert call; `0` behaves as `1` | +| `include_metadata` | bool | `true` | Include offset, timestamp, stream, topic and partition metadata | | `include_checksum` | bool | `true` | Include message checksum | | `include_origin_timestamp` | bool | `true` | Include client-provided timestamp | | `payload_format` | string | `"binary"` | How to store the payload: `"json"`, `"string"` (alias `"text"`), or `"binary"`; unknown values fall back to binary with a warning | -| `auto_create_collection` | bool | `false` | Create the collection if it doesn't exist | +| `auto_create_collection` | bool | `false` | Explicitly create a missing collection at startup; `false` still permits creation by the first insert | | `verbose_logging` | bool | `false` | Enable detailed logging | -| `max_retries` | u32 | `3` | Max retry attempts on failure | -| `retry_delay` | string | `"1s"` | Delay between retries | +| `max_retries` | u32 | `3` | Total attempts per transiently failing insert call, including the first; `0` and `1` both allow one attempt | +| `retry_delay` | string | `"1s"` | Base for linear backoff: delay multiplied by the retry number | + +## Stored Documents + +Every document has a generated `_id` of `stream:topic:partition:message_id` and a `payload` field. The offset is not part of the ID. Payload fields remain nested under `payload`, so a payload's own `_id` does not replace the connector's ID. Message headers are not stored. + +- `json` parses the payload and converts it to a BSON value under `payload`, including objects, arrays and scalars. Invalid JSON or an unsupported BSON conversion rejects the whole chunk before insertion. +- `string` and its `text` alias require valid UTF-8 and store a BSON string. +- `binary` stores BSON Binary with the generic subtype. Use `schema = "raw"` and no payload-changing transform to preserve original bytes. + +Format names are case-insensitive. Formatting happens after stream decoding and transforms, so JSON decoding can change the original bytes. + +With `include_metadata = true`, documents contain `iggy_offset`, `iggy_timestamp`, `iggy_stream`, `iggy_topic` and `iggy_partition_id`. Offsets above the signed 64-bit range use `iggy_offset_str` instead. Partition IDs use a BSON 32-bit integer when possible and a 64-bit integer otherwise. `iggy_checksum` is a 64-bit integer when possible and a decimal string otherwise. Message and origin timestamps become BSON datetimes, truncating microseconds to milliseconds. Checksum and origin timestamp flags are independent of `include_metadata`; `_id` and `payload` are always present. + +## Delivery Semantics + +Each polled batch is split into chunks of at most `batch_size` messages. Chunks are inserted immediately with unordered `insert_many`; other documents can succeed when one is rejected. Later chunks are still attempted after a failure, and the last chunk error is returned. There is no accumulation across polls and no transaction covering the batch. + +The connector retries transient insert failures up to `max_retries` total attempts, with linear waits of `retry_delay`, twice that delay, and so on. An invalid delay silently falls back to `1s`. Driver-level retryable writes may add retries independently, according to the URI and MongoDB deployment. Startup has no connector-level retry loop. + +An insert error containing only duplicate-key errors (`11000`) and no write-concern error is treated as success. This covers all unique indexes, not only `_id`, and does not compare payload contents. Reusing a message ID in one stream/topic/partition preserves the existing document. A conflict on another unique index can also discard a distinct message while runtime reports it processed. Account for this when choosing collection indexes. + +The runtime auto-commits while polling, records plugin errors and continues without replaying the failed batch. Earlier or later chunks may already be stored, and a timeout can leave the write outcome uncertain. Duplicate tolerance and retries do not provide an end-to-end at-least-once guarantee. ## Transforms diff --git a/content/docs/connectors/sinks/postgres.mdx b/content/docs/connectors/sinks/postgres.mdx index 723c98ef2e..fb386b74ef 100644 --- a/content/docs/connectors/sinks/postgres.mdx +++ b/content/docs/connectors/sinks/postgres.mdx @@ -18,13 +18,28 @@ This page is a curated subset of the documentation. The canonical reference, inc ## Configuration +Use the broker credentials and runtime setup from the [sink guide](/docs/connectors/sinks/sink#configuration). From the matching 0.9.0/edge repository root, build the plugin: + +```bash +cargo build --release -p iggy_connector_postgres_sink +``` + +For a local PostgreSQL instance matching the example credentials: + +```bash +docker run -d --name iggy-postgres -p 127.0.0.1:5432:5432 -e POSTGRES_USER=iggy -e POSTGRES_PASSWORD=iggy -e POSTGRES_DB=iggy postgres:15-alpine +docker exec iggy-postgres pg_isready -U iggy -d iggy +``` + +Wait for the readiness command to report accepting connections. Save this connector file in the runtime's connector directory. Replace the URI for your deployment. Startup opens the SQLx pool and runs `SELECT 1`; it creates the table only when requested. + ```toml type = "sink" key = "postgres-sink" enabled = true version = 1 name = "Postgres Sink" -path = "/path/to/libiggy_connector_postgres_sink.so" +path = "target/release/libiggy_connector_postgres_sink" [[streams]] stream = "user_events" @@ -35,9 +50,19 @@ poll_interval = "5ms" consumer_group = "postgres-sink" [plugin_config] -connection_string = "postgresql://username:password@localhost:5432/database" +connection_string = "postgresql://iggy:iggy@localhost:5432/iggy" target_table = "iggy_messages" auto_create_table = true +batch_size = 100 +payload_format = "bytea" +``` + +Create the Iggy resources and send a message: + +```bash +./target/release/iggy --username iggy --password iggy stream create user_events +./target/release/iggy --username iggy --password iggy topic create user_events events 1 none 1d +./target/release/iggy --username iggy --password iggy message send --partition-id 0 user_events events '{"user_id":"42","status":"active"}' ``` ### Plugin config options @@ -45,17 +70,17 @@ auto_create_table = true | Option | Type | Default | Description | | ------ | ---- | ------- | ----------- | | `connection_string` | string | required | PostgreSQL connection string | -| `target_table` | string | required | Table to insert messages into | -| `batch_size` | u32 | `100` | Messages per insert batch | +| `target_table` | string | required | One table identifier, quoted as a whole; `schema.table` names a literal table containing a dot | +| `batch_size` | u32 | `100` | Maximum messages per insert statement; `0` behaves as `1` | | `max_connections` | u32 | `10` | Max database connections | -| `auto_create_table` | bool | `false` | Create the target table if it doesn't exist | -| `include_metadata` | bool | `true` | Include Iggy metadata columns | +| `auto_create_table` | bool | `false` | Run `CREATE TABLE IF NOT EXISTS`; existing tables are not migrated | +| `include_metadata` | bool | `true` | Include offset, timestamp, stream, topic and partition columns | | `include_checksum` | bool | `true` | Include message checksum | | `include_origin_timestamp` | bool | `true` | Include original message timestamp | | `payload_format` | string | `"bytea"` | Payload column type: `bytea`, `json` (alias `jsonb`), or `text` | | `verbose_logging` | bool | `false` | Log at info level instead of debug | -| `max_retries` | u32 | `3` | Max retry attempts for transient errors | -| `retry_delay` | string | `"1s"` | Base delay between retries (e.g. `500ms`, `2s`) | +| `max_retries` | u32 | `3` | Total attempts per transiently failing insert, including the first; `0` and `1` both allow one attempt | +| `retry_delay` | string | `"1s"` | Base for linear retry delays; invalid values silently fall back to `1s` | ## Payload Format @@ -63,15 +88,17 @@ The `payload_format` option determines the type of the `payload` column and how | Format | Column Type | Description | | ------ | ----------- | ----------- | -| `bytea` | `BYTEA` | Raw bytes (default). Preserves exact binary content, works with any payload. | -| `json` / `jsonb` | `JSONB` | Native JSON. Enables JSON operators and GIN indexing. Payload must be valid JSON. | -| `text` | `TEXT` | UTF-8 text. Payload must be valid UTF-8. | +| `bytea` | `BYTEA` | Bytes after stream decoding and transforms (default). Use `schema = "raw"` and no payload-changing transform to preserve original bytes. | +| `json` / `jsonb` | `JSONB` | Native JSON. Enables JSON operators and GIN indexing. Payload must be valid JSON accepted by PostgreSQL JSONB. | +| `text` | `TEXT` | UTF-8 text. Payload must be valid UTF-8 accepted by PostgreSQL TEXT. | + +Format names are case-insensitive; unrecognized names silently use `bytea`. JSONB accepts objects, arrays and scalars, but PostgreSQL rejects `\u0000` in JSONB and zero bytes in TEXT. JSON decoding and JSONB storage can change the original representation. Message headers are not stored. See [PostgreSQL JSON types](https://www.postgresql.org/docs/15/datatype-json.html) for JSONB restrictions. -With the default `bytea` format, JSON payloads can still be queried by converting the bytes: `convert_from(payload, 'UTF8')::jsonb->>'user_id'`. With `json`, the native operators apply directly: `payload->>'user_id'`. +With the default `bytea` format, valid UTF-8 JSON payloads can still be queried by converting the bytes: `convert_from(payload, 'UTF8')::jsonb->>'user_id'`. With `json`, the native operators apply directly: `payload->>'user_id'`. ## Table Schema -When `auto_create_table` is enabled, the following table structure is created (the `payload` column type follows `payload_format`): +With `auto_create_table = true` and all three metadata flags enabled, the connector creates the following structure if the table is absent. The `payload` column type follows `payload_format`: ```sql CREATE TABLE iggy_messages ( @@ -88,8 +115,14 @@ CREATE TABLE iggy_messages ( ); ``` +Disabling `include_metadata` omits the five offset/timestamp/stream/topic/partition columns. Checksum and origin timestamp flags work independently. `id`, `payload` and `created_at` are always created. With `auto_create_table = false`, provision a compatible table yourself; startup does not check its existence or schema. + +`id` stores the full unsigned 128-bit message ID as a decimal, without stream, topic, partition or offset in the key. Offset and checksum values are cast to signed 64-bit integers, and partition IDs to signed 32-bit integers; values above those signed ranges appear negative. Message and origin timestamps are interpreted as Unix microseconds. Zero becomes the Unix epoch; a timestamp outside the date library's range falls back to the current time. `created_at` is generated by PostgreSQL when inserting. + ## Querying the Data +Run these queries with `docker exec -it iggy-postgres psql -U iggy -d iggy`, or your PostgreSQL client: + ```sql -- Get all messages from a specific stream SELECT * FROM iggy_messages WHERE iggy_stream = 'user_events'; @@ -102,8 +135,17 @@ WHERE iggy_stream = 'user_events'; For a JSONB query example, a TEXT search example, and recommended indexes, see the upstream [postgres_sink README](https://github.com/apache/iggy/tree/master/core/connectors/sinks/postgres_sink). +## Delivery Semantics + +Each poll is split into chunks of at most `batch_size` messages. One multi-row `INSERT` writes each chunk; a duplicate primary key, incompatible table, invalid payload or other terminal error rejects that chunk. Later chunks are still attempted. The plugin logs failures, increments its private insertion-error count by the failed chunk size, and returns success after attempting all chunks. Both its processed count and runtime's processed count can therefore include messages that were never stored. Runtime error counts do not expose these insert failures. + +Transient insert failures are retried within the chunk. `max_retries` counts total attempts, and waits grow linearly as `retry_delay`, twice that delay, and so on. Retried SQLSTATEs are `40001`, `40P01`, `57P01`, `57P02`, `57P03`, `08000`, `08003` and `08006`; SQLx I/O and pool-acquisition timeouts are also retried. Other errors stop that chunk's retry loop. Startup has no plugin retry loop. + +The sink uses plain `INSERT`, with no conflict-ignore or upsert clause. Reusing an ID already in the table rejects the entire chunk, including any new IDs alongside it. There is no transaction covering the complete poll. The runtime auto-commits while polling and does not replay failed chunks; a connection failure can leave the write outcome uncertain. These retries do not provide an end-to-end at-least-once guarantee. + ## Performance Considerations - Use an appropriate `batch_size` for your workload (larger batches give better throughput) - Create indexes on frequently queried columns (`iggy_stream`, `iggy_topic`, `created_at`) -- Monitor connection pool usage via `max_connections` +- Set the pool limit with `max_connections`; it does not report pool usage +- Keep each actual chunk within PostgreSQL's [65,535 query-parameter limit](https://www.postgresql.org/docs/15/limits.html). Each row binds `2 + 5 * include_metadata + include_checksum + include_origin_timestamp` parameters, so the default flags allow at most 7,281 rows per statement. Larger chunks are rejected by SQLx; `batch_length` also limits how many rows reach a poll. diff --git a/content/docs/connectors/sinks/quickwit.mdx b/content/docs/connectors/sinks/quickwit.mdx index 3d177e26fb..d572c3f9dd 100644 --- a/content/docs/connectors/sinks/quickwit.mdx +++ b/content/docs/connectors/sinks/quickwit.mdx @@ -3,14 +3,41 @@ title: Quickwit Sink description: "Send messages from Iggy streams to a Quickwit index over HTTP, creating the index when it does not exist." --- -The Quickwit connector allows you to send data to the Quickwit API using HTTP. This sink will ensure that the index exists (create it if it doesn't) and will append the data to the index using the same batch size as specified in the Iggy configuration. +The Quickwit connector allows you to send data to the Quickwit API using HTTP. At startup it checks readiness, creates a missing index, and appends messages as newline-delimited JSON (NDJSON). Each poll can become multiple requests, each limited to 8 MiB including document newlines. ## Configuration -- `url`: The URL of the Quickwit server. -- `index`: The index configuration using YAML, as described in the [Quickwit index configuration docs](https://quickwit.io/docs/configuration/index-config) +Use the broker credentials and runtime setup from the [sink guide](/docs/connectors/sinks/sink#configuration). From the matching 0.9.0/edge Iggy checkout root, build the plugin: + +```bash +cargo build --release -p iggy_connector_quickwit_sink +``` + +For a local Quickwit instance with support for the example's 0.9 index configuration: + +```bash +docker run -d --name iggy-quickwit -p 127.0.0.1:7280:7280 -e QW_LISTEN_ADDRESS=0.0.0.0 quickwit/quickwit:edge run +curl --fail http://localhost:7280/health/readyz +``` + +Wait for readiness, then save this connector file in the runtime's connector directory. `index` is a YAML string following the [Quickwit index configuration format](https://quickwit.io/docs/configuration/index-config). Use a format version supported by your Quickwit server. The sink sends this YAML when creating a missing index; it does not update or compare an existing index's mapping. ```toml +type = "sink" +key = "quickwit" +enabled = true +version = 1 +name = "Quickwit Sink" +path = "target/release/libiggy_connector_quickwit_sink" + +[[streams]] +stream = "events" +topics = ["logs"] +schema = "json" +batch_length = 100 +poll_interval = "5ms" +consumer_group = "quickwit-sink" + [plugin_config] url = "http://localhost:7280" index = """ @@ -63,3 +90,62 @@ retention: schedule: daily """ ``` + +Create the Iggy resources and send a document that matches the strict mapping: + +```bash +./target/release/iggy --username iggy --password iggy stream create events +./target/release/iggy --username iggy --password iggy topic create events logs 1 none 1d +timestamp=$(date +%s) +./target/release/iggy --username iggy --password iggy message send --partition-id 0 events logs "{\"timestamp\":$timestamp,\"service_name\":\"example\",\"message\":\"hello Quickwit\"}" +``` + +Indexing happens asynchronously. Search after the split has committed: + +```bash +curl --fail --get http://localhost:7280/api/v1/events/search --data-urlencode 'query=message:hello' +``` + +`commit_timeout_secs = 10` requests a time-based commit; it is not a ten-second delivery deadline. Retention uses the document's `timestamp` field. The example requests seven-day retention checked daily. The timestamp mapping accepts Unix timestamps and formats search results as nanoseconds, with millisecond fast-field precision. + +## Plugin config options + +| Option | Default | Description | +| ------ | ------- | ----------- | +| `url` | required | HTTP(S) base URL with a host; path prefixes/trailing slashes work, query strings and fragments are rejected | +| `index` | required | YAML index configuration containing a nonempty `index_id` | +| `verbose_logging` | `false` | Log received/submitted document counts at info instead of debug | +| `max_retries` | `3` | Total HTTP attempts including the first; `0` and `1` both allow one attempt | +| `retry_delay` | `"1s"` | Base exponential delay for HTTP retries and readiness probes | +| `retry_max_delay` | `"5s"` | Cap for calculated HTTP retry delays; a valid `Retry-After` on HTTP 429 overrides it | +| `max_open_retries` | `10` | Total readiness attempts including the first; `0` and `1` both allow one attempt | +| `open_retry_max_delay` | `"30s"` | Cap for calculated readiness retry delays | +| `timeout` | `"30s"` | Timeout per HTTP attempt; retries and their waits can make an operation take longer | + +Durations require units and must be positive; invalid or zero durations prevent startup. Unknown plugin keys are rejected. There is no plugin `batch_size` option: `batch_length` limits the polled batch, and the sink splits it by serialized byte size without accumulating across polls. + +Readiness uses `GET /health/readyz`, retrying any failed probe. The index check uses `GET /api/v1/indexes/`; only a 404 triggers creation with `POST /api/v1/indexes`. If creation receives an error status, a successful index recheck allows startup to continue. A successful existence check does not validate the supplied mapping against the existing one. Requests retain any path prefix from `url`. + +## Payload Shapes + +The sink does not add Iggy IDs, offsets, timestamps or headers. It serializes the payload after runtime decoding and transforms: + +| Payload | Document sent to Quickwit | +| ------- | ------------------------- | +| JSON object, or a JSON object parsed from raw bytes | The object itself | +| JSON array or scalar | `{"data":[1,2],"data_type":"json"}` or `{"data":42,"data_type":"json"}` | +| Other raw UTF-8 | `{"data":"hello","data_type":"raw","data_encoding":"utf8"}` | +| Raw non-UTF-8 bytes | Base64 under `data`, with `data_type = "raw"` and `data_encoding = "base64"` | +| Text | `{"text":"hello","data_type":"text"}` | + +Raw arrays, scalars and malformed JSON retain their original bytes in the raw wrapper. Avro and FlatBuffer payload variants use that raw path; Proto uses the text wrapper. Which variant arrives depends on the runtime decoder and transforms. + +The strict mapping above accepts only mapped fields and requires `timestamp`. To use wrapper documents, map their fields or use `mode: dynamic`, and remove `timestamp_field` and its dependent `retention` section unless every document supplies a timestamp. The `add_fields` transform enriches JSON objects; it does not add timestamps to raw/text wrappers created later by this sink. + +## Delivery Semantics + +The sink calls `POST /api/v1//ingest?commit=auto`. HTTP success means submission for indexing, not search visibility or acceptance of every document. The sink checks the status and does not inspect per-document rejection counts in the response. See the [Quickwit ingest API](https://quickwit.io/docs/reference/rest-api#ingest-api). + +HTTP 429, 5xx and network failures can retry. Calculated waits use exponential backoff with jitter and the configured cap; a valid `Retry-After` on 429 replaces that wait. Other HTTP statuses stop that request's retry loop. Readiness has its separate attempt budget. There is no circuit breaker or sink deduplication key, so retrying an accepted request can produce duplicates. `max_retries = 1` disables the sink's HTTP retry loop. + +An individual serialized document exceeding 8 MiB, including its newline, is logged and skipped. Other documents and later chunks are still attempted, and the last error is returned. Successful chunks are not rolled back. The runtime logs/counts plugin callback errors and continues without replaying the failed batch; it auto-commits while polling. A failed callback can therefore coexist with stored documents, and a successful callback can include documents rejected by Quickwit. Runtime processed counts do not prove successful indexing. The sink provides no end-to-end at-least-once guarantee. diff --git a/content/docs/connectors/sinks/s3.mdx b/content/docs/connectors/sinks/s3.mdx index c4fb11205e..3513a55cce 100644 --- a/content/docs/connectors/sinks/s3.mdx +++ b/content/docs/connectors/sinks/s3.mdx @@ -9,6 +9,14 @@ This page is a curated subset of the documentation. The canonical reference is t ## Configuration +Use the broker credentials and runtime setup from the [sink guide](/docs/connectors/sinks/sink#configuration). From the matching 0.9.0/edge Iggy checkout root, build the plugin: + +```bash +cargo build --release -p iggy_connector_s3_sink +``` + +Create the bucket separately, then save this connector file in the runtime's connector directory. The plugin does not create buckets. Paths below assume the runtime starts from the checkout root. + ```toml type = "sink" key = "s3" @@ -41,27 +49,50 @@ output_format = "json_lines" | Option | Type | Default | Description | | ------ | ---- | ------- | ----------- | | `bucket` | string | **required** | S3 bucket name | -| `region` | string | **required** | AWS region, e.g. `us-east-1` (`auto` for R2) | -| `prefix` | string | none | Key prefix prepended to all objects | +| `region` | string | **required** | AWS region, e.g. `us-east-1` (`auto` for [R2](https://developers.cloudflare.com/r2/api/s3/api/)) | +| `prefix` | string | none | Prefix for data objects and loss markers; leading/trailing slashes are removed | | `endpoint` | string | none | Custom S3 endpoint for MinIO, R2, and other compatible stores | | `path_template` | string | `{stream}/{topic}/{date}/{hour}` | Directory structure for S3 keys | | `file_rotation` | string | `size` | `size` or `messages` | -| `max_file_size` | string | `8MiB` | Rotation threshold when `file_rotation = "size"` | -| `max_messages_per_file` | u64 | none | Rotation threshold, required when `file_rotation = "messages"` | +| `max_file_size` | string | `8MiB` | Size rotation threshold; must be positive and at most `5GiB`, even in message-count mode | +| `max_messages_per_file` | u64 | none | Positive count required in `messages` mode; ignored by size rotation | | `output_format` | string | `json_lines` | `json_lines`, `json_array`, or `raw` | -| `include_metadata` | bool | `true` | Include stream/topic/partition/offset in the output | -| `include_headers` | bool | `false` | Include message headers in the output | +| `include_metadata` | bool | `true` | Include stream/topic/partition/offset/timestamp in JSON output | +| `include_headers` | bool | `false` | Include available headers in JSON output, independently of metadata | -Further options cover credentials (`access_key_id` / `secret_access_key`), retries (`max_attempts` `3`, `retry_delay` `1s` with exponential backoff and jitter), and `path_style` addressing (auto-enabled when `endpoint` is set). See the upstream [s3_sink README](https://github.com/apache/iggy/tree/master/core/connectors/sinks/s3_sink) for the full reference. +Further options cover credentials (`access_key_id` / `secret_access_key`), retries (`max_attempts` defaults to `3`, `retry_delay` to `1s`), and `path_style` addressing (auto-enabled when `endpoint` is set, overridable with `false`). `max_retries` is an alias for `max_attempts`; `0` and `1` both make one outer upload attempt. Invalid retry durations prevent startup; `0s` is allowed. Output format names are case-insensitive, with `jsonl` and `jsonlines` aliases for `json_lines`. See the upstream [s3_sink README](https://github.com/apache/iggy/tree/master/core/connectors/sinks/s3_sink) for the full reference. -Path templates support `{stream}`, `{topic}`, `{partition}`, `{date}`, `{hour}`, and `{timestamp}`. The time-based variables derive from the first message timestamp in each buffer. File names embed the partition (zero-padded to 5 digits, to prevent cross-partition collisions) and the offset range (zero-padded to 20 digits), e.g. `application_logs/api_requests/2026-03-16/14/00000-00000000000000000000-00000000000000000999.jsonl`. +Each stream/topic/partition has its own buffer, retained across polls. Rotation is checked after appending a complete formatted record. Size mode counts record bytes but excludes JSON newlines, commas and array brackets, so an object can exceed `max_file_size`. Message-count mode does not also enforce the size threshold. Uploads use one `PutObject`, with no multipart upload; choose thresholds and message sizes that fit the [store's single-upload limit](https://docs.aws.amazon.com/AmazonS3/latest/userguide/upload-objects.html). There is no timer flush: a partial buffer waits until its threshold is reached or the connector closes. + +Path templates support `{stream}`, `{topic}`, `{partition}`, `{date}`, `{hour}`, and `{timestamp}`. The time-based variables derive from the first message timestamp in each buffer: date/hour use UTC, and `{timestamp}` is epoch milliseconds. File names embed the partition (zero-padded to 5 digits, to prevent cross-partition collisions) and the offset range (zero-padded to 20 digits), e.g. `application_logs/api_requests/2026-03-16/14/00000-00000000000000000000-00000000000000000999.jsonl` before adding the configured prefix. + +Stream/topic substitutions preserve ASCII letters, digits, `.`, `_` and `-`, replacing other characters with `_`. Distinct names can therefore produce the same key. Keep stream/topic separation in the template, use names that remain distinct after replacement, and avoid sharing the same output paths across independent producers. A repeated object key overwrites the current object, subject to bucket versioning; the offset range alone does not provide global deduplication. Extensions are `.jsonl`, `.json` and `.bin` for the three formats. ## Credentials -Credentials resolve in order of precedence: explicit `access_key_id` + `secret_access_key` in the config (both together or neither), the standard `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` / `AWS_SESSION_TOKEN` environment variables, the AWS shared credentials file (`~/.aws/credentials`), then container credentials or the instance profile when running on EC2/ECS/EKS. +Credentials resolve in order of precedence: explicit `access_key_id` + `secret_access_key` in the config (both together or neither), environment credentials, the shared credentials file, STS web identity, ECS container credentials, then EC2 instance metadata (IMDSv2 before v1). The chain is provided by the pinned `aws-creds` dependency, not the AWS SDK. + +Environment credentials use `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`, with optional `AWS_SESSION_TOKEN` / `AWS_SECURITY_TOKEN`. The file path is `AWS_SHARED_CREDENTIALS_FILE` or `~/.aws/credentials`; this plugin reads its `[default]` section and does not select `AWS_PROFILE`. Web identity uses `AWS_ROLE_ARN` and `AWS_WEB_IDENTITY_TOKEN_FILE`. The container path uses `AWS_CONTAINER_CREDENTIALS_RELATIVE_URI`. Explicit config credentials have no session-token option; use the environment or credentials file for a temporary key pair and token. + +Startup writes an empty object at the bucket-root key `.iggy-sink-probe`, then attempts to delete it. The probe ignores `prefix`, needs `PutObject` on that key, and rejects non-success HTTP statuses. `ListBucket` is not required; failed probe cleanup does not prevent startup. Reserve this key: an existing object at that name is overwritten, and deleted if the credentials permit cleanup. ### MinIO example +Start a local MinIO server: + +```bash +docker run -d --name iggy-s3 -p 127.0.0.1:9000:9000 -e MINIO_ROOT_USER=minioadmin -e MINIO_ROOT_PASSWORD=minioadmin minio/minio:RELEASE.2025-09-07T16-13-09Z server /data +curl --fail http://localhost:9000/minio/health/live +``` + +After the health request succeeds, create the bucket using MinIO's client. These commands use Docker host networking on Linux: + +```bash +docker run --rm --network host -e MC_HOST_iggy=http://minioadmin:minioadmin@localhost:9000 minio/mc mb iggy/my-bucket +``` + +Replace the earlier `[plugin_config]` table with this local configuration. One message per file makes the example upload immediately: + ```toml [plugin_config] bucket = "my-bucket" @@ -69,11 +100,43 @@ region = "us-east-1" endpoint = "http://localhost:9000" access_key_id = "minioadmin" secret_access_key = "minioadmin" +file_rotation = "messages" +max_messages_per_file = 1 +output_format = "json_lines" +``` + +Create the Iggy resources, send a JSON message, and start the runtime as described in the sink guide: + +```bash +./target/release/iggy --username iggy --password iggy stream create application_logs +./target/release/iggy --username iggy --password iggy topic create application_logs api_requests 1 none 1d +./target/release/iggy --username iggy --password iggy topic create application_logs errors 1 none 1d +./target/release/iggy --username iggy --password iggy message send --partition-id 0 application_logs api_requests '{"method":"GET","path":"/api/users","status":200}' +``` + +List the uploaded objects: + +```bash +docker run --rm --network host -e MC_HOST_iggy=http://minioadmin:minioadmin@localhost:9000 minio/mc ls --recursive iggy/my-bucket ``` +## Output Formats + +JSON Lines and JSON Array contain wrapper objects with the decoded/transformed payload under `payload`. JSON payloads retain their JSON value; text and Proto variants become strings. Raw bytes containing valid JSON are parsed, while other raw bytes, Avro and FlatBuffer variants become base64 strings. Those strings do not carry an encoding tag. + +With metadata enabled, each wrapper also contains `offset`, a UTC timestamp at second precision, `stream`, `topic` and `partition_id`. Message IDs, checksums and origin timestamps are not included. Headers are a separate optional object: string values remain strings, raw values become base64, booleans become booleans and supported numeric types become JSON numbers; other values use strings. + +Raw output concatenates payload bytes without delimiters and ignores both inclusion flags. To preserve the original payload bytes, use `schema = "raw"` and no transform that changes them; decoding JSON before raw output can reserialize it. + ## Delivery Semantics -The connector runtime commits consumer offsets before `consume()` runs and does not inspect its return value, so the effective guarantee is **at-most-once**: upload failures are retried only by the sink's internal retry loop, and a crash loses whatever is buffered in memory but not yet flushed to S3 (there is no write-ahead log or dead-letter queue). Size your rotation thresholds accordingly: smaller files bound the loss window, larger files upload more efficiently. +The runtime auto-commits while polling. It logs/counts a failed plugin callback and continues without replaying that batch. A successful callback can mean only that messages were buffered, so runtime processed counts do not establish delivery to S3. A crash loses unflushed buffers; there is no write-ahead log or dead-letter queue. + +The sink retries HTTP 408, 429, 5xx and client errors. Other HTTP statuses stop the upload immediately. Its exponential backoff includes jitter and a 60-second cap, without handling `Retry-After`. The S3 library also retries transport errors once internally and uses a 60-second request timeout, so `max_attempts` counts outer `PutObject` calls rather than individual network requests or a total time budget. The startup probe does not use the sink's upload retry loop. + +Buffers are cleared before upload. A failed upload loses those messages and stops processing the remainder of that poll. The sink tries to write an additional `.lost` object containing the offset range, message count and error, using the same retry policy. This marker contains no payload and may also fail. Graceful close attempts to flush remaining buffers, logs failures, and still returns success. Neither runtime counters nor shutdown success prove that every message was stored, and the sink provides no end-to-end delivery or deduplication guarantee. + +Size your rotation thresholds accordingly: smaller files bound the loss window, larger files upload more efficiently. ## Transforms diff --git a/content/docs/connectors/sinks/sink.mdx b/content/docs/connectors/sinks/sink.mdx index 2359c39761..d5127aee3b 100644 --- a/content/docs/connectors/sinks/sink.mdx +++ b/content/docs/connectors/sinks/sink.mdx @@ -10,6 +10,9 @@ Sink connectors are responsible for writing data from Iggy streams to external s The sink is represented by the single `Sink` trait, which defines the basic interface for all sink connectors. It provides methods for initializing the sink, writing data to external destination, and closing the sink. ```rust +use async_trait::async_trait; +use iggy_connector_sdk::{ConsumedMessage, Error, MessagesMetadata, TopicMetadata}; + #[async_trait] pub trait Sink: Send + Sync { /// Invoked when the sink is initialized, allowing it to perform any necessary setup. @@ -30,32 +33,36 @@ pub trait Sink: Send + Sync { ## Configuration -Each sink connector is configured in its own separate configuration file within the connectors directory specified in the main runtime config. - -```rust -pub struct SinkConfig { - pub key: String, - pub enabled: bool, - pub version: u64, - pub name: String, - pub path: String, - pub transforms: Option, - pub streams: Vec, - pub plugin_config_format: Option, - pub plugin_config: Option, - pub verbose: bool, - pub benchmark: bool, -} -``` +With the local configuration provider, each sink connector has a TOML file in the directory specified by the main runtime config. The [runtime](/docs/connectors/runtime#configuration-providers) also supports an HTTP configuration provider. + +| Field | Meaning | +|-------|---------| +| `key` | Connector key (`String`), separate from the numeric plugin instance ID. | +| `enabled` | Whether to start the connector (`bool`). | +| `version` | Configuration version (`u64`). | +| `name` | Display name (`String`). | +| `path` | Shared-library path (`String`). | +| `streams` | Stream configuration entries. | +| `transforms` | Optional transform configuration. | +| `plugin_config` | Optional custom configuration object deserialized by the plugin. | +| `plugin_config_format` | Optional default format for the HTTP API's plugin-config response. The FFI always receives JSON. | +| `verbose`, `benchmark` | Optional logging flags (`bool`). | The two flags at the end are optional and default to `false`: `verbose` switches the connector's per-batch logging to info level, and `benchmark` emits per-batch timing events for performance measurement. -**Main runtime config (config.toml):** +**Main runtime config (connectors.toml, in the repository root):** + +Use the matching 0.9.0/edge checkout and broker from the [quick start](/docs/connectors/introduction#quick-start). Its broker must use the credentials below. Create the `connectors` directory, then save these two configuration files there and in the repository root as labeled. ```toml +[iggy] +address = "localhost:8090" +username = "iggy" +password = "iggy" + [connectors] config_type = "local" -config_dir = "path/to/connectors" +config_dir = "connectors" ``` **Sink connector config (connectors/stdout.toml):** @@ -63,7 +70,7 @@ config_dir = "path/to/connectors" ```toml # Type of connector (sink or source) type = "sink" -key = "stdout" # Unique sink ID +key = "stdout" # Unique sink key # Required configuration for a sink connector enabled = true @@ -99,31 +106,43 @@ value.static = "hello" ### Environment Variable Overrides -Configuration properties can be overridden using environment variables. The pattern follows: `IGGY_CONNECTORS_SINK_[KEY]_[PROPERTY]` +Configuration properties can be overridden using environment variables. Supported scalar fields and indexed stream entries use the pattern: `IGGY_CONNECTORS_SINK_[KEY]_[PROPERTY]` -For example, to override the `enabled` property for a sink with ID `stdout`: +For example, to disable the sink with key `stdout` at the next runtime start: ```bash -IGGY_CONNECTORS_SINK_STDOUT_ENABLED=false +export IGGY_CONNECTORS_SINK_STDOUT_ENABLED=false ``` -Top-level fields of `plugin_config` can be overridden (or injected) the same way with the `IGGY_CONNECTORS_SINK_[KEY]_PLUGIN_CONFIG_[FIELD]` pattern. This is the recommended way to pass credentials without writing them into the TOML file: +Top-level fields of `plugin_config` can be overridden (or injected) the same way with the `IGGY_CONNECTORS_SINK_[KEY]_PLUGIN_CONFIG_[FIELD]` pattern. This lets you supply credentials without writing them into the TOML file: ```bash -IGGY_CONNECTORS_SINK_STDOUT_PLUGIN_CONFIG_PRINT_PAYLOAD=true +export IGGY_CONNECTORS_SINK_STDOUT_PLUGIN_CONFIG_PRINT_PAYLOAD=true ``` ## Sample implementation Let's implement the example sink connector, which will simply print the messages to the standard output. -Additionally, our sink connector will have its own state, which can be used e.g. to track the overall progress or store some relevant information when ingesting the data further to the external sources or tooling. +This sink keeps an in-memory invocation counter. The runtime does not persist sink plugin state; Iggy consumer-group offsets are managed separately. Also, when implementing the sink connector, make sure to use the `sink_connector!` macro to expose the FFI interface and allow the connector runtime to register the sink with the runtime. And finally, each sink should have its own, custom configuration, which is passed along with the unique plugin ID via expected `new()` method. -Let's start by defining the internal state and the public sink connector along with its own configuration. +Use the [reference crate's Cargo.toml](https://github.com/apache/iggy/blob/master/core/connectors/sinks/stdout_sink/Cargo.toml) from the matching checkout, including `[lib] crate-type = ["cdylib", "lib"]` and its workspace dependency features. The Rust blocks through the `Sink` implementation below combine into `src/lib.rs`; the trait above is a separate API reference. + +Start with the imports, then define the state, connector and configuration. + +```rust +use async_trait::async_trait; +use iggy_connector_sdk::{ + ConsumedMessage, Error, MessagesMetadata, Sink, TopicMetadata, sink_connector, +}; +use serde::{Deserialize, Serialize}; +use tokio::sync::Mutex; +use tracing::info; +``` ```rust #[derive(Debug)] @@ -137,7 +156,7 @@ struct State { pub struct StdoutSink { id: u32, print_payload: bool, - state: Mutex + state: Mutex, } ``` @@ -166,7 +185,7 @@ We can invoke the expected macro to expose the FFI interface and allow the conne sink_connector!(StdoutSink); ``` -At a bare minimum, we need to add the following dependencies to the `Cargo.toml` file to compile the plugin at all: +The reference manifest supplies these dependencies: - async-trait - dashmap @@ -230,16 +249,17 @@ impl Sink for StdoutSink { It's also important to note, that the supported format(s) might vary depending on the connector implementation. For example, you might expect `JSON` as the payload format, which can be then easily parsed and processed by upstream components such as data transforms, but at the same time, you could support the other formats and let the user decide which one to use. -For example, you can match against the `payload` enum field containing the deserialized value to process (or not) the consumed message(s). +For a sink that only prints JSON, replace the `if self.print_payload` block in `consume()` with `print_json_messages(messages, &messages_metadata);` and add this function. Unsupported payloads are logged and skipped by this variant: ```rust -for message in messages { - match message.payload { - Payload::Json(value) => { - // Process JSON payload - } - _ => { - warn!("Unsupported payload format: {}", messages_metadata.schema); +use iggy_connector_sdk::Payload; +use tracing::warn; + +pub fn print_json_messages(messages: Vec, metadata: &MessagesMetadata) { + for message in messages { + match message.payload { + Payload::Json(value) => info!("JSON payload: {value}"), + _ => warn!("Unsupported payload format: {}", metadata.schema), } } } @@ -247,9 +267,28 @@ for message in messages { While the schema of messages (that will be consumed from the Iggy stream), cannot be controlled by the sink connector itself, the built-in configuration allows to decide what's the expected format of the messages (the particular `StreamDecoder` will be used). -Keep in mind, that it might be sometimes difficult/impossible e.g. to transform one format to another e.g. JSON to SBE or so, and in such a case, the consumed messages will be ignored. +The runtime logs and counts decoding or transform failures, drops the affected messages and passes the remaining batch to the plugin. A plugin `consume()` error is counted as a failed batch. The runtime uses auto-commit when polling and does not automatically retry that failed batch; any destination retry must be implemented by the plugin. + +Build the matching plugin, runtime and CLI from the repository root: + +```bash +cargo build --release -p iggy_connector_stdout_sink -p iggy-connectors -p iggy-cli +``` + +On an empty broker, create the configured stream and topic: + +```bash +./target/release/iggy --username iggy --password iggy stream create example_stream +./target/release/iggy --username iggy --password iggy topic create example_stream example_topic 1 none 1d +``` + +After saving the configuration files above, start the runtime. This explicitly enables the sink if you tried the optional disabling override: + +```bash +IGGY_CONNECTORS_SINK_STDOUT_ENABLED=true IGGY_CONNECTORS_CONFIG_PATH=connectors.toml cargo run --release --bin iggy-connectors +``` -Eventually, compile the source code and create a separate connector configuration file in the connectors directory (as specified in the main runtime `config.toml`). Make sure that `path` points to the existing plugin. +Produce JSON messages to this topic, for example with the [Random source](/docs/connectors/sources/random), to see the sink output. And that's all, enjoy using the sink connector! diff --git a/content/docs/connectors/sinks/stdout.mdx b/content/docs/connectors/sinks/stdout.mdx index 0fea9c2317..8a853f3591 100644 --- a/content/docs/connectors/sinks/stdout.mdx +++ b/content/docs/connectors/sinks/stdout.mdx @@ -5,15 +5,23 @@ description: "Print messages from Iggy streams to standard output, for debugging The stdout sink connector prints messages from Iggy streams to the standard output. This is useful for debugging, development, and verifying that your connector pipeline is working correctly. +The connector emits info-level tracing events through the runtime to stdout. The runtime selects text or JSON with `logging.format` and filters events with `RUST_LOG` (default `INFO`). It logs batch metadata on every invocation; `print_payload` additionally logs each message's offset and decoded payload. The invocation counter is in memory and resets when the plugin is recreated. + ## Configuration +Use the broker credentials and runtime setup from the [sink guide](/docs/connectors/sinks/sink#configuration). Save the configuration below in that runtime's connector directory. From the matching 0.9.0/edge repository root, build the plugin: + +```bash +cargo build --release -p iggy_connector_stdout_sink +``` + ```toml type = "sink" key = "stdout-sink" enabled = true version = 1 name = "Stdout Sink" -path = "/path/to/libiggy_connector_stdout_sink.so" +path = "target/release/libiggy_connector_stdout_sink" verbose = false [[streams]] @@ -33,3 +41,5 @@ print_payload = true | Option | Type | Default | Description | |--------|------|---------|-------------| | `print_payload` | bool | `false` | Whether to print the message payload to stdout | + +Create `my-stream` and `my-topic` before starting the runtime, or change the sample to use existing resources. The [Random source](/docs/connectors/sources/random) configuration produces JSON to the same destination. diff --git a/content/docs/connectors/sinks/surrealdb.mdx b/content/docs/connectors/sinks/surrealdb.mdx index 0ec515d7e0..1446bc92bc 100644 --- a/content/docs/connectors/sinks/surrealdb.mdx +++ b/content/docs/connectors/sinks/surrealdb.mdx @@ -1,14 +1,29 @@ --- title: SurrealDB Sink -description: "Write messages from Iggy streams into SurrealDB over the HTTP API, one bulk insert per batch." +description: "Write messages from Iggy streams into SurrealDB over the HTTP API, bulk inserts split by the configured batch size." --- -The SurrealDB sink connector writes messages from Iggy streams into SurrealDB over the HTTP API. Each batch becomes one SurrealQL bulk `INSERT IGNORE`. Every record gets a deterministic record id derived from stream, topic, partition, offset, and message id, so replayed batches are idempotent and existing records are left untouched. +The SurrealDB sink connector writes messages from Iggy streams into SurrealDB over the HTTP API. Each poll is split into chunks of at most `batch_size` messages, with one SurrealQL bulk `INSERT IGNORE ... RETURN NONE` for each chunk containing valid records. Every record gets a deterministic record id derived from stream, topic, partition, offset, and message id, so replaying the same identities leaves existing records untouched. Messages are not buffered across polls. This page is a curated subset of the documentation. The canonical reference is the upstream [surrealdb_sink README](https://github.com/apache/iggy/tree/master/core/connectors/sinks/surrealdb_sink) in the `apache/iggy` repository. ## Configuration +Use the broker credentials and runtime setup from the [sink guide](/docs/connectors/sinks/sink#configuration). From the matching 0.9.0/edge Iggy checkout root, build the plugin: + +```bash +cargo build --release -p iggy_connector_surrealdb_sink +``` + +Start a local SurrealDB instance with the credentials used below. This example stores SurrealDB data in memory: + +```bash +docker run -d --name iggy-surrealdb -p 127.0.0.1:8000:8000 surrealdb/surrealdb:v3.1.4 start --bind 0.0.0.0:8000 --user root --pass root memory +curl --fail http://127.0.0.1:8000/health +``` + +After the health request succeeds, save this connector file in the runtime's connector directory. Its plugin path assumes the runtime starts from the checkout root. + ```toml type = "sink" key = "surrealdb" @@ -37,6 +52,20 @@ auto_define_table = true batch_size = 1000 ``` +Create the Iggy resources and send a JSON message, then start the runtime as described in the sink guide: + +```bash +./target/release/iggy --username iggy --password iggy stream create example_stream +./target/release/iggy --username iggy --password iggy topic create example_stream example_topic 1 none 1d +./target/release/iggy --username iggy --password iggy message send --partition-id 0 example_stream example_topic '{"event":"created","count":1}' +``` + +Query the stored records: + +```bash +curl --fail --user root:root -H 'Accept: application/json' -H 'Surreal-NS: iggy' -H 'Surreal-DB: connectors' --data 'SELECT * FROM iggy_messages;' http://127.0.0.1:8000/sql +``` + ### Common Options | Option | Type | Default | Description | @@ -45,24 +74,45 @@ batch_size = 1000 | `namespace` | string | **required** | Namespace selected at startup | | `database` | string | **required** | Database selected at startup | | `table` | string | **required** | Target table; must be a safe SurrealQL identifier | -| `username` / `password` | string | none | Optional credentials | +| `username` / `password` | string | none | Both required unless `auth_scope = "none"` | | `auth_scope` | string | `root` | `root`, `namespace`, `database`, or `none` | | `use_tls` | bool | `false` | Use `https://` when `endpoint` has no scheme | -| `auto_define_table` | bool | `false` | Run `DEFINE TABLE IF NOT EXISTS SCHEMALESS` at startup | -| `define_indexes` | bool | `false` | Define an offset index on stream/topic/partition/offset; requires `auto_define_table` | -| `batch_size` | u32 | `1000` | Maximum records per SurrealDB request | +| `auto_define_table` | bool | `false` | Create missing namespace/database and a schemaless table; requires `auth_scope = "root"` | +| `define_indexes` | bool | `false` | Define a non-unique stream/topic/partition/offset index; requires metadata and only runs with automatic table definition | +| `batch_size` | u32 | `1000` | Maximum input records per insert chunk; `0` is raised to `1` | | `payload_format` | string | `auto` | `auto`, `json`, `text`, `base64`, or `binary` (alias for `base64`) | | `include_metadata` | bool | `true` | Store stream/topic/partition/offset/timestamp/schema fields | -The `include_headers`, `include_checksum`, and `include_origin_timestamp` flags also default to `true`. Retry and timeout options: `query_timeout` (`30s`), `max_retries` (`3` total attempts), `retry_delay` (`100ms`), `max_retry_delay` (`5s`), plus `verbose_logging`. See the upstream [surrealdb_sink README](https://github.com/apache/iggy/tree/master/core/connectors/sinks/surrealdb_sink) for the full table. +The `include_headers`, `include_checksum`, and `include_origin_timestamp` flags also default to `true` and work independently of `include_metadata`. Retry and timeout options are `query_timeout` (`30s` per HTTP request), `max_retries` (`3` total write attempts, with `0` raised to `1`), `retry_delay` (`100ms`), and `max_retry_delay` (`5s`). Invalid duration strings log a warning and fall back to `1s`; zero durations are accepted. The maximum delay is raised to the base delay if configured smaller. `verbose_logging` defaults to `false` and raises per-poll submission logs from debug to info. See the upstream [surrealdb_sink README](https://github.com/apache/iggy/tree/master/core/connectors/sinks/surrealdb_sink) for the full table. + +Namespace, database and table names must start with an ASCII letter or underscore and contain only ASCII letters, digits and underscores. Endpoint URLs cannot contain credentials, a path beyond `/`, a query or a fragment. An explicit `http://` or `https://` scheme takes precedence over `use_tls`. Payload-format and authentication-scope names are case-insensitive; unknown values prevent startup. + +With root, namespace or database authentication, startup calls `/signin` and SQL requests use Basic authentication with the configured scope. Namespace/database users require pre-existing resources and `auto_define_table = false`. With `auth_scope = "none"`, credentials are ignored; the server must permit those unauthenticated operations. Automatic DDL uses `IF NOT EXISTS` and does not migrate existing definitions. The optional `
_iggy_offset_idx` index is not a uniqueness constraint. `define_indexes = true` with metadata disabled rejects startup; without automatic DDL it only logs a warning and skips index creation. ## Stored Shape -With metadata enabled, each record contains the deterministic `id`, the `iggy_message_id`, provenance fields (`iggy_stream`, `iggy_topic`, `iggy_partition_id`, `iggy_offset`, `iggy_timestamp`, `iggy_origin_timestamp`, `iggy_checksum`, `iggy_schema`), `iggy_headers`, the `payload`, and its `payload_encoding`. With `payload_format = "auto"`, JSON payloads are stored as queryable SurrealDB values, text payloads as strings, and binary payloads as base64 strings. +Every submitted record contains `id`, `iggy_message_id`, `payload` and `payload_encoding`. `include_metadata` adds `iggy_stream`, `iggy_topic`, `iggy_partition_id`, `iggy_offset`, `iggy_timestamp` and `iggy_schema`. The checksum and origin timestamp have separate flags. `iggy_headers` appears only when enabled and nonempty. + +Iggy message IDs, partition IDs, offsets, timestamps and checksums are stored as strings. Timestamps retain Iggy's microsecond values, not SurrealDB datetime values. Record IDs encode stream/topic UTF-8 bytes as hex, then append partition, offset and the 32-digit hexadecimal message ID. Changing the payload or transforms does not change that identity; changing any identity component creates a different key. + +Payload formatting operates after the runtime decoder and transforms: + +| `payload_format` | Stored payload and encoding | +| ---------------- | --------------------------- | +| `auto` | JSON values as `json`; text/Proto variants as `text`; raw/Avro/FlatBuffer bytes as `base64`, even when raw bytes contain valid JSON | +| `json` | A JSON value, parsing other payload variants as JSON; invalid JSON is rejected | +| `text` | A string, decoding other payload bytes as UTF-8; invalid UTF-8 is rejected | +| `base64` / `binary` | Base64 of the payload bytes, with `payload_encoding = "base64"` | + +Non-raw headers are stored as strings, including numbers and booleans. Raw headers use `{"data":"AQID","iggy_header_encoding":"base64"}`. Payload JSON is sent as SurrealQL values, so destination schema, numeric and other value constraints still apply. ## Delivery Semantics -The runtime commits consumer offsets before `consume()` runs, so persistent write failures are **at-most-once**: they are logged but not redelivered. Transient failures are retried inside the batch with capped exponential backoff. Because records use deterministic ids and `INSERT IGNORE`, a replayed batch (for example after a connector restart re-reads uncommitted messages) doesn't create duplicates. +The runtime auto-commits while polling, logs/counts plugin callback failures, and continues without replaying the failed batch. The sink retries transaction-conflict errors, connection/timeouts, and HTTP 408, 429, 500, 502, 503 and 504. Delays use capped exponential backoff with jitter; `Retry-After` is not used. Connection errors trigger reconnection and repeat sign-in/health/optional DDL. A failed reconnect stops that chunk, and retries plus startup requests can exceed `query_timeout` in total. + +A malformed record is skipped while valid records in its chunk are still submitted. A failed chunk does not stop later chunks, and the last error is returned. The sink checks HTTP and SQL statement statuses. Its own processed counter counts records submitted in successful statements, including records ignored by `INSERT IGNORE`; runtime counters instead reflect whether the whole callback succeeded. Neither is a count of newly inserted rows. + +[SurrealQL `INSERT IGNORE`](https://surrealdb.com/docs/reference/query-language/statements/insert#ignoring-duplicates) preserves existing records with the same ID, including their old payloads. With the SurrealDB 3.1.4 instance above, `IGNORE` also silently skips unique-index conflicts and field-assertion failures while accepting valid records from the same chunk. These skips return a successful statement and are included in processed counters. This protects repeated writes of the same identities; it does not ensure that a failed batch will be delivered. There is no end-to-end at-least-once guarantee. ## Transforms diff --git a/content/docs/connectors/sources/elasticsearch.mdx b/content/docs/connectors/sources/elasticsearch.mdx index 63f53882c4..85c1518ad3 100644 --- a/content/docs/connectors/sources/elasticsearch.mdx +++ b/content/docs/connectors/sources/elasticsearch.mdx @@ -3,10 +3,32 @@ title: Elasticsearch Source description: "Poll documents from an Elasticsearch index into Iggy streams, incrementally when a timestamp field is configured." --- -The Elasticsearch source connector polls documents from an Elasticsearch index and produces them to Iggy streams. With a timestamp field configured, each poll only fetches documents newer than the last one seen, giving incremental, restart-safe ingestion. +The Elasticsearch source connector polls documents from an Elasticsearch index and produces them to Iggy streams. With a timestamp field configured, it filters searches using the last acknowledged timestamp. This is a timestamp watermark, with the limitations described below. ## Configuration +Use the broker credentials and main runtime configuration from the [source guide](/docs/connectors/sources/source#configuration). From the matching 0.9.0/edge Iggy checkout root, build the plugin: + +```bash +cargo build --release -p iggy_connector_elasticsearch_source +``` + +For this local example, start Elasticsearch with authentication disabled: + +```bash +docker run -d --name iggy-elasticsearch-source -p 127.0.0.1:9200:9200 -e discovery.type=single-node -e xpack.security.enabled=false -e 'ES_JAVA_OPTS=-Xms512m -Xmx512m' elasticsearch:9.3.0 +curl --fail 'http://localhost:9200/_cluster/health?wait_for_status=yellow&timeout=30s' +``` + +Wait until the health response reports `timed_out: false`, then create the source index and a document: + +```bash +curl --fail -X PUT http://localhost:9200/logs -H 'Content-Type: application/json' --data '{"mappings":{"properties":{"@timestamp":{"type":"date"},"level":{"type":"keyword"}}}}' +curl --fail -X PUT 'http://localhost:9200/logs/_doc/1?refresh=true' -H 'Content-Type: application/json' --data '{"@timestamp":"2026-01-01T00:00:00Z","level":"error","message":"example"}' +``` + +Save this connector entry in the runtime's connector directory. Start the runtime from the checkout root so the plugin path resolves. + ```toml type = "source" key = "elasticsearch" @@ -30,17 +52,30 @@ batch_size = 100 timestamp_field = "@timestamp" ``` +Create the destination before starting the runtime: + +```bash +./target/release/iggy --username iggy --password iggy stream create elasticsearch_stream +./target/release/iggy --username iggy --password iggy topic create elasticsearch_stream documents 1 none 1d +``` + +Start the runtime as described in the source guide. After the configured 30-second polling delay, read the produced message: + +```bash +./target/release/iggy --username iggy --password iggy message poll --offset 0 elasticsearch_stream documents 0 +``` + ### Plugin config options | Option | Type | Default | Description | | ------ | ---- | ------- | ----------- | | `url` | string | required | Elasticsearch cluster URL | | `index` | string | required | Index to poll; must exist when the connector starts | -| `username` / `password` | string | none | Optional basic authentication credentials | +| `username` / `password` | string | none | Basic authentication is enabled only when both are present | | `query` | table | `match_all` | Elasticsearch query DSL applied on each poll | -| `polling_interval` | string | `"10s"` | Delay before each poll cycle | -| `batch_size` | integer | `100` | Maximum documents fetched per poll | -| `timestamp_field` | string | none | Document field used for incremental polling | +| `polling_interval` | string | `"10s"` | Delay before each poll cycle; invalid strings fall back to `10s`, and zero is accepted | +| `batch_size` | integer | `100` | Search size per poll; `0` returns no hits, and Elasticsearch enforces its own result-window limit | +| `timestamp_field` | string | none | Top-level RFC3339 string field used to advance the watermark | A custom `query` must be a structured object. In TOML that means nested tables (an inline JSON string won't work): @@ -49,16 +84,26 @@ A custom `query` must be a structured object. In TOML that means nested tables ( value = "error" ``` -Alternatively, set `plugin_config_format = "json"` and provide the plugin config as JSON, as the crate's own [config example](https://github.com/apache/iggy/tree/master/core/connectors/sources/elasticsearch_source) does. +The connector file remains TOML. `plugin_config_format = "json"` only selects the default format of the HTTP API's plugin-config response; it does not change how the local file is parsed. The [upstream config example](https://github.com/apache/iggy/tree/master/core/connectors/sources/elasticsearch_source) also uses TOML with a structured `[plugin_config]` table. ## How Polling Works -Each cycle the connector waits `polling_interval`, then runs a search against `index` with the configured `query`, `size = batch_size`, sorted ascending by `timestamp_field` (`@timestamp` when unset). Every document's `_source` becomes one JSON message. +Each cycle the connector waits `polling_interval`, then runs a search against `index` with the configured `query`, `size = batch_size`, sorted ascending by `timestamp_field` (`@timestamp` when unset). Every hit with `_source` becomes one JSON message. Elasticsearch `_id` is not copied into the payload or used as the Iggy message ID; headers and timestamps are also unset by this plugin. -When `timestamp_field` is set, the query is also filtered to documents whose timestamp is greater than the newest one seen so far, so already-processed documents are not fetched again. Without a `timestamp_field` there is no incremental cursor and each poll returns the first `batch_size` matching documents. +When `timestamp_field` is set, the query is also filtered to values strictly greater than the last acknowledged timestamp. Only top-level strings that parse as RFC3339 update the watermark; numeric timestamps, date-only strings and nested paths do not. Without a `timestamp_field` there is no incremental cursor and each poll returns the first `batch_size` matching documents. + +The connector does not use scroll, `search_after` or a document-ID tiebreaker. If a timestamp group spans multiple batches, advancing past that timestamp skips the remaining tied documents. Late arrivals and updates at or below the watermark are also skipped. `scroll_timeout` is accepted but unused. Without a usable timestamp, repeated polls can produce duplicates; the default `@timestamp` sort still requires a sortable field in the index mapping. + +Search and JSON-decoding failures, `timed_out: true`, and a nonzero `_shards.failed` count are returned as poll errors without advancing the watermark. The next polling cycle retries from the previous watermark. Poll errors are logged by the SDK; they do not increment the runtime's forwarding-error counter or change the runtime connector status. There is no per-query retry loop or configured HTTP request timeout. The source stages a candidate watermark and applies it only after runtime `Ack`; `Nack` keeps the previous watermark so the next search can fetch the rejected records again. Fetched-document and byte counters include these repeated attempts. This does not remove the timestamp limitations above or provide exactly-once delivery. ## State -State is managed by the connectors runtime, not by the connector itself. Along with each batch of produced messages, the connector returns its progress (last poll timestamp plus document, poll, and error counters). The runtime persists it atomically to a per-connector file, `source_.state`, under the `[state]` path from the runtime config (default `local_state`). On restart the runtime hands the saved state back to the connector, which resumes polling from the last processed timestamp. +Along with each successful poll, including an empty one, the connector returns a MessagePack checkpoint containing its timestamp watermark and counters. After sending the batch, the runtime saves that checkpoint and acknowledges it. With the default file state backend, it uses `source_.state` under the runtime `[state]` path (default `local_state`); the runtime also supports an HTTP state backend. On restart, the saved checkpoint is passed to the plugin. Invalid MessagePack state logs a warning and starts fresh. + +Runtime state persistence requires no plugin configuration. To reset its position with the file backend, stop the runtime and delete the source's state file. + +The optional `[plugin_config.state]` enables a separate JSON snapshot loaded during `open()` and saved during `close()`. Its loaded values can override the runtime checkpoint. It defaults to `./connector_states/elasticsearch_source_.json`; `state_id` and `storage_config.base_path` customize that location. Only file storage is implemented. Selecting `elasticsearch`, `redis` or an unknown storage type warns and falls back to `./connector_states`, ignoring the supplied backend location. + +`state.auto_save_interval` and `state.tracked_fields` do not affect the runtime plugin. Its JSON snapshot is a direct file write, separate from the runtime's atomic checkpoint protocol, and snapshot errors only warn. If enabled, account for this second snapshot when resetting or restoring progress; removing only the runtime state file does not reset the plugin snapshot. -State persistence requires no plugin configuration. To reset the connector's position, stop the runtime and delete the state file. +The saved counters include fetched documents, successful polls (including empty polls), empty polls, serialized payload bytes, errors and the last error text. The average successful-poll duration includes the configured polling delay. Document-ID, scroll-ID and offset fields are retained in state but do not drive polling. These plugin counters are separate from the runtime's sent/error metrics. diff --git a/content/docs/connectors/sources/influxdb.mdx b/content/docs/connectors/sources/influxdb.mdx index ac0a2879cc..1af18083c3 100644 --- a/content/docs/connectors/sources/influxdb.mdx +++ b/content/docs/connectors/sources/influxdb.mdx @@ -3,12 +3,20 @@ title: InfluxDB Source description: "Poll InfluxDB on an interval and produce the resulting rows into an Iggy stream, on both InfluxDB V2 and V3." --- -The InfluxDB source connector polls InfluxDB on an interval and produces the resulting rows as messages into an Iggy stream. It supports both InfluxDB V2 (OSS 2.x / Cloud, Flux queries) and InfluxDB V3 (Core / Enterprise, SQL queries), selected with the `version` option. A timestamp cursor stored in persistent connector state tracks the position, so restarts resume where they left off. +The InfluxDB source connector polls InfluxDB on an interval and produces the resulting rows as messages into an Iggy stream. It supports both InfluxDB V2 (OSS 2.x / Cloud, Flux queries) and InfluxDB V3 (Core / Enterprise, SQL queries), selected with the `version` option. An acknowledged timestamp cursor and row offset are stored in connector state. Restarts resume from that checkpoint, subject to the ordering and late-arrival limits below. This page is a curated subset of the documentation. The canonical reference, including the full cursor semantics and stuck-timestamp handling, is the upstream [influxdb_source README](https://github.com/apache/iggy/tree/master/core/connectors/sources/influxdb_source) in the `apache/iggy` repository. ## Configuration +Use the broker credentials and main runtime configuration from the [source guide](/docs/connectors/sources/source#configuration). Build the plugin from the matching 0.9.0/edge checkout root: + +```bash +cargo build --release -p iggy_connector_influxdb_source +``` + +The V2 example requires an existing `telemetry` bucket in `iggy_org`, a token with read access, and `cpu` points with a numeric `usage` field and a `host` tag. Save this entry in the runtime's connector directory and start the runtime from the checkout root. + ```toml type = "source" key = "influxdb" @@ -32,15 +40,23 @@ token = "replace_with_secret_token" query = ''' from(bucket: "telemetry") |> range(start: time(v: "$cursor")) - |> filter(fn: (r) => r._measurement == "cpu") - |> sort(columns: ["_time"]) + |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage") + |> group(columns: []) + |> sort(columns: ["_time", "host"]) |> limit(n: $limit) ''' poll_interval = "5s" batch_size = 500 ``` -InfluxDB V2 takes an `org` and a Flux query. V3 takes a `db` and a SQL query: +Create the destination before starting the runtime: + +```bash +./target/release/iggy --username iggy --password iggy stream create events +./target/release/iggy --username iggy --password iggy topic create events influx_events 1 none 1d +``` + +InfluxDB V2 takes an `org` and a Flux query. For V3, replace the entire `[plugin_config]` section below, including the V2-only `org`. The `my-db` database must contain a `cpu` table; this example assumes `host` is its only tag. ```toml # V3 variant @@ -52,12 +68,18 @@ token = "replace_with_secret_token" query = ''' SELECT * FROM cpu WHERE time > '$cursor' -ORDER BY time +ORDER BY time, host LIMIT $limit OFFSET $offset ''' ``` -Omitting `version` defaults to `"v2"` for backward compatibility. The query template must contain the `$cursor` placeholder (enforced at startup). `$limit` is strongly recommended so `batch_size` bounds each query. V3 queries also need `OFFSET $offset` unless stuck-batch detection is disabled. +Omitting the plugin `version` defaults to `"v2"` for backward compatibility; it is separate from the connector entry's numeric configuration version. Unknown plugin keys and unsupported versions are rejected. The query must contain `$cursor` outside comments. Keep `$limit` in the query so the connector can control its size. With stuck-batch detection enabled, V3 also requires `$offset` and an ascending `ORDER BY`. Startup checks use text matching, so they do not prove that a query implements the required ordering. + +Start the runtime as described in the source guide. After a polling cycle, read the produced records: + +```bash +./target/release/iggy --username iggy --password iggy message poll --offset 0 events influx_events 0 +``` ### Common Options @@ -67,27 +89,42 @@ Omitting `version` defaults to `"v2"` for backward compatibility. The query temp | `url` | string | **required** | InfluxDB base URL | | `org` | string | **required** (v2) | Organization name | | `db` | string | **required** (v3) | Target database | -| `token` | string | **required** | API token, never logged | -| `query` | string | **required** | Flux (v2) or SQL (v3) query template with `$cursor` / `$limit` placeholders | -| `poll_interval` | string | `5s` | How often to issue queries | -| `batch_size` | u32 | `500` | Maximum rows per query | +| `token` | string | **required** | API token; its debug representation is redacted | +| `query` | string | **required** | Flux (v2) or SQL (v3) query template | +| `poll_interval` | string | `5s` | Delay before each polling cycle | +| `batch_size` | u32 | `500` | Base query size; `0` is clamped to `1`, and cursor handling can enlarge `$limit` | | `cursor_field` | string | `_time` (v2), `time` (v3) | Column used as the cursor | -| `initial_offset` | string | none | Starting cursor timestamp on first run | +| `initial_offset` | string | `1970-01-01T00:00:00Z` | Starting RFC3339 cursor when no checkpoint exists | | `payload_format` | string | `json` | `json`, `text`, or `raw` | -| `include_metadata` | bool | `true` | Include all row columns in the payload | -| `stuck_batch_cap_factor` | u32 | `10` | V3 only: cap on batch inflation for tied timestamps (max `100`, `0` disables) | +| `include_metadata` | bool | `true` | Controls columns in the whole-row JSON payload; see below | +| `stuck_batch_cap_factor` | u32 | `10` | V3 only: inflation cap; accepts `2` through `100`, or `0` to disable the guards | + +`payload_column` extracts one column. Without it, every `payload_format` setting emits whole-row JSON. With it, the selected `json`, `text` or `raw` format applies, and the destination `streams.schema` should match. Selecting a column that is missing from a row causes a poll error. -Further options cover `payload_column` (extract a single column as the payload, required for `text` and `raw` formats) and resilience: `timeout` (`10s`), `max_retries` (`3`), `retry_delay` / `retry_max_delay`, startup health-check retries, and a circuit breaker (`circuit_breaker_threshold` `5`, `circuit_breaker_cool_down` `30s`). See the upstream [influxdb_source README](https://github.com/apache/iggy/tree/master/core/connectors/sources/influxdb_source) for the full list. +Resilience defaults are `timeout = "10s"`, `max_retries = 3`, `retry_delay = "1s"`, `retry_max_delay = "5s"`, `max_open_retries = 10`, `open_retry_max_delay = "60s"`, `circuit_breaker_threshold = 5`, and `circuit_breaker_cool_down = "30s"`. Attempt counts include the first attempt and are clamped to at least one. Query retries cover network errors, HTTP 429 and 5xx, with exponential backoff and jitter; an integer-seconds `Retry-After` on 429 can override the delay cap. Other HTTP failures and malformed successful responses return poll errors. Invalid duration strings warn and fall back to `1s`; zero durations are accepted. `verbose_logging` defaults to `false`. ## Cursor-Based Polling -On each poll the cursor (an RFC 3339 timestamp) is substituted into the query template. After a successful batch it advances to the highest timestamp seen. The two versions differ deliberately: +The connector substitutes its RFC3339 cursor into the query. Both versions require ascending timestamps and a stable, unique order within each timestamp group. Include all necessary tag or key columns after the timestamp in the sort. The examples use `host` because their data model has one point per host and timestamp. -- **V2** uses inclusive `>=` semantics, so the query must sort by the cursor field (`|> sort(columns: ["_time"])`). Startup fails for `>=`-style queries lacking a sort. `range()`-style queries pass that check without a sort, but should keep it anyway, since the skip count below depends on ordered rows. Rows at the cursor timestamp are re-fetched after an advance, and the connector skips exactly the already-delivered ones using a persisted row count, preventing duplicates. -- **V3** uses strict `WHERE time > '$cursor'`, so no rows are re-delivered across batches. Because InfluxDB 3's query engine has no stable order for rows sharing one timestamp, the connector pages through tied rows with `OFFSET $offset`, inflating the batch size up to `stuck_batch_cap_factor` times `batch_size` before tripping the circuit breaker. +- **V2** uses an inclusive cursor, such as `range(start: time(v: "$cursor"))`. Flux sorts and limits each table separately, so the example groups its selected numeric field into one table before sorting. A persisted count skips rows already acknowledged at the cursor timestamp. `$limit` is the base batch size plus that count, capped at eleven times the base size. Dense timestamp groups can exhaust the skip allowance and produce a stuck-cursor error without advancing the cursor. A strict `>` query can lose unseen ties and is unsuitable for this skip-count scheme. +- **V3** requires strict `WHERE time > '$cursor'`; inclusive `>=` templates are rejected. A full batch containing one timestamp retains the previous cursor, advances `$offset`, and doubles the next query size up to the configured cap. A full batch containing several timestamps defers the final timestamp group to the next poll. Reaching the inflation cap emits no messages, resets the effective size to the base size, preserves the offset, and records a circuit-breaker failure. The breaker opens only when its failure threshold is reached. Setting the cap factor to `0` disables these guards and can skip unseen tied rows. + +A timestamp watermark is not change-data capture. Late inserts, edits and backdated points at or below an acknowledged watermark can be missed, and changing the order of tied rows invalidates offset-based progress. Neither version tracks deletions. The connector's timestamp-plus-position message IDs can also collide for distinct rows across batches; they are not unique database-row keys. ## Payload Formats -- **`json`** (default): each row becomes a JSON object. V3 emits flat rows with native JSON types. V2 wraps each row in an envelope (`measurement`, `field`, `timestamp`, `value`, `row`) with all values as strings coerced to bool/int/float where possible, since annotated CSV carries no types. A string field containing `"42"` arrives as the number `42` under V2 (coerced) but stays the string `"42"` under V3. Values that can't be coerced stay strings under V2. Migrating consumers must update deserialization. -- **`text`**: the value of `payload_column` is produced as a UTF-8 string. -- **`raw`**: the value of `payload_column` is base64-decoded and produced as raw bytes. +- **Whole-row `json`**: V3 emits a flat object with native JSON types; `include_metadata = false` removes only the cursor column. V2 emits an envelope with `measurement`, `field`, `timestamp`, `value` and `row`. Its parser reads CSV cells as strings and ignores the `#datatype` and `#default` values. Payload conversion tries bool, integer and finite float, then string; an empty cell becomes null. Thus a string field containing `"42"` becomes the number `42` in V2 but stays a string in V3. With metadata disabled, V2's `row` retains only `_time` and `_value`; the envelope remains. +- **Selected-column `json`**: V2 parses the cell as JSON. V3 serializes the column's existing JSON value, so a string containing JSON remains a JSON string. +- **Selected-column `text`**: V2 emits the cell as UTF-8. V3 emits strings directly and serializes other JSON values as text. +- **Selected-column `raw`**: the column must contain standard base64 text, which is decoded into raw bytes. + +Missing selected columns, invalid JSON cells in V2, and invalid base64 reject the poll. V3 also rejects any row with a missing, non-string or invalid cursor. V2 can emit rows without valid cursors if another row supplies a usable watermark; keep the cursor column in every row. V3 timestamps without a timezone suffix are treated as UTC, preserving nanoseconds. Keep database timestamps in the payload when consumers need them; the runtime does not transfer the plugin's timestamp fields into broker message metadata. + +## State and Failures + +Successful polls return a versioned MessagePack checkpoint, including empty polls. After forwarding messages, the runtime saves it and sends `Ack`; only then does the plugin apply its candidate cursor and row count. `Nack` discards that candidate so the next poll can fetch the records again. A failure after forwarding but before acknowledgement can produce duplicates. + +The default file backend stores `source_influxdb.state` under the runtime's state path (default `local_state`). The runtime also supports an HTTP state backend. V2 restores versioned state or its legacy unversioned state; V3 requires versioned V3 state. Corrupt checkpoints, invalid saved timestamps and mismatched versions prevent startup. To reset file-backed progress, stop the runtime and remove the source's checkpoint, then restart with the desired `initial_offset`. + +Query and parsing failures leave the acknowledged cursor unchanged and are logged as poll errors. They do not increment the runtime's forwarding-error counter or change its running status. While its circuit breaker is open, the plugin waits and returns empty batches without checkpoints. On cooldown expiry it permits another query. A V3 response reporting that the database was not found is treated as an empty result, so create the database before starting the source. diff --git a/content/docs/connectors/sources/postgres.mdx b/content/docs/connectors/sources/postgres.mdx index ebef4c8e76..343735b66c 100644 --- a/content/docs/connectors/sources/postgres.mdx +++ b/content/docs/connectors/sources/postgres.mdx @@ -13,19 +13,47 @@ This page is a curated subset of the documentation. The canonical reference, inc - **Change Data Capture**: Monitor database changes using PostgreSQL logical replication (`test_decoding` plugin) - **Payload Column Extraction**: Emit a single column directly as the message payload (raw bytes, text, or JSONB) - **Custom Queries**: Use custom SQL with parameter substitution instead of simple table polling -- **Delete / Mark Processed**: Optionally delete rows after reading or flag them in a boolean column +- **Delete / Mark Processed**: Optionally delete rows or flag them in a boolean column after acknowledgement - **Offset Tracking**: Per-table tracking offsets are persisted as connector state and survive restarts -- **Automatic Retries**: Transient database errors are retried with a configurable delay +- **Automatic Retries**: Transient polling-query errors are retried with a configurable delay ## Configuration +Use the broker credentials and main runtime configuration from the [source guide](/docs/connectors/sources/source#configuration). Build the plugin from the matching 0.9.0/edge checkout root: + +```bash +cargo build --release -p iggy_connector_postgres_source +``` + +The example uses an existing PostgreSQL database named `iggy`, with an `iggy` user and password. Run this SQL in that database: + +```sql +CREATE TABLE users ( + id BIGINT PRIMARY KEY, + name TEXT NOT NULL, + email TEXT NOT NULL, + updated_at TIMESTAMPTZ NOT NULL +); +CREATE TABLE orders ( + id BIGINT PRIMARY KEY, + user_id BIGINT NOT NULL, + description TEXT NOT NULL +); +INSERT INTO users VALUES + (123, 'John Doe', 'john@example.com', '2024-01-15T10:29:50Z'); +INSERT INTO orders VALUES (1, 123, 'Example order'); +``` + +Save this entry in the runtime's connector directory and start the runtime from the checkout root: + ```toml type = "source" key = "postgres-source" enabled = true version = 1 name = "Postgres Source" -path = "/path/to/libiggy_connector_postgres_source.so" +path = "target/release/libiggy_connector_postgres_source" +plugin_config_format = "toml" [[streams]] stream = "database_changes" @@ -35,12 +63,25 @@ batch_length = 100 linger_time = "5ms" [plugin_config] -connection_string = "postgresql://username:password@localhost:5432/database" +connection_string = "postgresql://iggy:iggy@localhost:5432/iggy" mode = "polling" tables = ["users", "orders"] poll_interval = "30s" batch_size = 1000 -tracking_column = "updated_at" +tracking_column = "id" +``` + +Create the destination before starting the runtime: + +```bash +./target/release/iggy --username iggy --password iggy stream create database_changes +./target/release/iggy --username iggy --password iggy topic create database_changes table_events 1 none 1d +``` + +After a polling cycle, read the produced records: + +```bash +./target/release/iggy --username iggy --password iggy message poll --offset 0 database_changes table_events 0 ``` ### Plugin config options @@ -49,26 +90,40 @@ tracking_column = "updated_at" | ------ | ---- | ------- | ----------- | | `connection_string` | string | required | PostgreSQL connection string | | `mode` | string | required | `polling` or `cdc` | -| `tables` | array | required | Tables to monitor | -| `poll_interval` | string | `"10s"` | How often to poll (e.g. `1s`, `5m`) | -| `batch_size` | u32 | `1000` | Max rows per poll | -| `tracking_column` | string | `"id"` | Column for incremental updates | -| `initial_offset` | string | none | Starting value for the tracking column | +| `tables` | array | required | Polling tables; an empty list polls none. In CDC, an empty list captures all tables | +| `poll_interval` | string | `"10s"` | Delay before each cycle (e.g. `1s`, `5m`); invalid values fall back to `10s` | +| `batch_size` | u32 | `1000` | Polling: limit per table, with `0` returning no rows. CDC: a transaction-boundary limit that a transaction can exceed | +| `tracking_column` | string | `"id"` | Column used by the strict `>` polling watermark | +| `initial_offset` | string | none | Exclusive starting value when a table has no saved offset | | `max_connections` | u32 | `10` | Max database connections | | `snake_case_columns` | bool | `false` | Convert column names to snake_case | -| `include_metadata` | bool | `true` | Wrap results with metadata | +| `include_metadata` | bool | `true` | Polling envelope layout; `false` currently nests the row under `data.data` and retains metadata | | `payload_column` | string | none | Column to extract directly as the message payload | -| `payload_format` | string | none | Format of `payload_column`: `bytea` (alias `raw`), `text`, or `json_direct` (alias `jsonb`) | -| `delete_after_read` | bool | `false` | Delete rows after reading | -| `processed_column` | string | none | Boolean column to mark rows as processed | +| `payload_format` | string | none | Defaults to `json`; selected-column formats are described below | +| `delete_after_read` | bool | `false` | Delete selected rows after Ack; takes precedence over marking | +| `processed_column` | string | none | Adds a `FALSE` filter to the default query and marks selected rows after Ack | | `primary_key_column` | string | `tracking_column` | Primary key used for delete/mark operations | | `custom_query` | string | none | Custom SQL with parameter substitution | | `replication_slot` | string | `"iggy_slot"` | Replication slot name (CDC mode only) | | `capture_operations` | array | `["INSERT","UPDATE","DELETE"]` | CDC operations to capture | | `cdc_backend` | string | `"builtin"` | CDC backend; only `builtin` is implemented | | `verbose_logging` | bool | `false` | Log at info level instead of debug | -| `max_retries` | u32 | `3` | Max retry attempts for transient errors | -| `retry_delay` | string | `"1s"` | Base delay between retries | +| `max_retries` | u32 | `3` | Total polling-query attempts, including the first; `0` still makes one attempt | +| `retry_delay` | string | `"1s"` | Linear delay: this duration times the failed attempt number; invalid values fall back to `1s` | + +## Polling and Delivery + +The default query selects rows with `tracking_column > last_offset`, orders that column ascending, and applies `batch_size` separately to each table. Use a non-null, unique tracking value that increases in the database's sort order. A timestamp with ties can skip rows at a batch boundary, and inserts or updates at or below the watermark are not revisited. Polling does not capture deletions. + +A `custom_query` replaces the entire default query, including its ordering, limit and processed-row filter. Its placeholders are `$table`, `$offset`, `$limit`, `$now` (RFC3339 UTC) and `$now_unix` (integer seconds). Substitution is textual, with no automatic quoting or escaping; `$offset` is empty if neither saved state nor `initial_offset` supplies it. Keep queries trusted and explicitly preserve ascending tracking-column order. The offset comes from the last returned row with a usable tracking value. + +Polling stages its cursor and processed-row count until the runtime forwards the batch, saves its checkpoint and sends `Ack`. `Nack` discards that candidate so the next poll can read the rows again. A failure after forwarding can therefore duplicate messages. IDs are random UUIDs for each poll, so replayed rows do not retain a stable message ID. + +`delete_after_read` and `processed_column` stage cleanup keys during polling and modify PostgreSQL only after `Ack`. A Nack or failed poll leaves the rows available for replay. Cleanup is separate from delivery and checkpointing: a cleanup error stops the source, and a crash after checkpointing but before cleanup can leave already-delivered rows unmodified in PostgreSQL. Cleanup across tables is not atomic. + +The file state backend stores `source_postgres-source.state` under the runtime's state path (default `local_state`). Its MessagePack state contains per-table offsets, a last-poll time and a processed-row count. The HTTP state backend is also available. Missing or undecodable plugin state starts fresh; file access errors prevent startup. Stop the runtime before resetting its checkpoint. `initial_offset` is used only where saved offsets are absent. + +The plugin's retries cover transient polling `SELECT` failures, including connection I/O, pool timeouts, serialization failures and deadlocks. Connection setup, CDC fetches and delete/mark statements do not use that retry loop. A poll error is logged and polling continues without incrementing the runtime's forwarding-error counter. Repeated delivery or checkpoint failures instead receive `Nack`; the SDK stops after five consecutive Nacks. ## Output Format @@ -93,15 +148,23 @@ The stream config should use `schema = "json"`. Flat-schema sinks such as Iceber ### Payload Column Extraction -When `payload_column` is set, the connector skips the envelope and emits that column directly as the message payload. `payload_format` controls how the column is read: `bytea` passes raw bytes through (use `schema = "raw"`), `text` reads UTF-8 text, and `json_direct` serializes a JSONB column to JSON bytes. See the upstream README for schema pairing and round-trip recipes with the PostgreSQL sink. +In polling mode, an existing `payload_column` bypasses the envelope. Use `bytea` or `raw` for a BYTEA column with `schema = "raw"`, `text` for a text column with `schema = "text"`, and `json_direct`, `jsonb` or `jsonb_direct` for JSON/JSONB with `schema = "json"`. A null BYTEA/text payload becomes empty bytes, which the Iggy message builder rejects; the entire batch is Nacked. Use non-null, non-empty BYTEA/text payloads. A null JSON/JSONB payload becomes JSON `null` and can be delivered. + +The default `payload_format = "json"` reads a selected column as BYTEA but reports JSON to the runtime, so those bytes must contain valid JSON. Without a selected column, the connector always emits a JSON envelope. If the selected column is absent, it falls back to the envelope while retaining the selected format's schema; use a column present in every configured table. Type mismatches reject the poll. See the upstream README for sink round-trip recipes. + +With no selected column, `include_metadata = false` currently retains the envelope and adds an extra `data` level. Whole-row BYTEA fields are base64 strings. Keep database timestamps in the payload when needed; the runtime does not copy the plugin's timestamp fields into broker message metadata. ## CDC Mode With `mode = "cdc"` the connector reads changes from a logical replication slot instead of polling tables: - The builtin backend creates (or reuses) a logical replication slot using the `test_decoding` output plugin. The default slot name is `iggy_slot`. No publication is created or required, since `test_decoding` ignores publications entirely. -- PostgreSQL must run with `wal_level = logical`, and the connection must be direct (no pooler) where replication requires it. -- `capture_operations` filters which operations (`INSERT`, `UPDATE`, `DELETE`) are emitted. +- PostgreSQL must run with `wal_level = logical` and have `test_decoding` installed and allowed. The login needs access to the logical-decoding SQL functions. This backend uses ordinary SQL connections, not a replication-protocol connection; any proxy must support its SQL and slot operations. +- `capture_operations` accepts uppercase `INSERT`, `UPDATE` and `DELETE`; an empty array emits none. Filtering happens after reading the slot. Qualified table names match the schema and table; unqualified names match that table name in any schema. An empty `tables` array captures all tables. - The `pg_replicate` backend (based on Supabase's ETL framework) is **not implemented**: selecting `cdc_backend = "pg_replicate"` fails at startup unless the `cdc_pg_replicate` build feature is enabled, and the backend returns an error even when it is. -When decommissioning a CDC connector, drop the replication slot to stop WAL retention. See the upstream README for details. +Each connector needs its own slot. Reusing one slot across connectors divides its changes between them. `pg_logical_slot_get_changes` consumes slot progress before delivery, including changes excluded by the filters. A later failure cannot replay those changes from the slot, and the plugin checkpoint does not contain a recovery LSN. + +CDC always builds a JSON envelope with the operation and parsed columns. Do not combine it with polling payload extraction options. Deletes contain replica-identity columns; updates may also supply `old_data` from an `old-key` tuple. Quoted `test_decoding` values, including JSONB and arrays, remain strings. Unchanged TOAST values are represented as null, which does not establish that the database value is null. Envelope timestamps are generated while polling, not PostgreSQL transaction commit times. + +When decommissioning a CDC connector, stop it and drop its replication slot to stop WAL retention. With PostgreSQL's default `max_slot_wal_keep_size = -1`, an unused slot can retain unbounded WAL. See the upstream README for details. diff --git a/content/docs/connectors/sources/random.mdx b/content/docs/connectors/sources/random.mdx index a4e052a279..bc8a62e879 100644 --- a/content/docs/connectors/sources/random.mdx +++ b/content/docs/connectors/sources/random.mdx @@ -7,13 +7,19 @@ The random source connector generates random messages and sends them to Iggy str ## Configuration +Use the broker credentials and runtime setup from the [source guide](/docs/connectors/sources/source#configuration). Save the configuration below in that runtime's connector directory. From the matching 0.9.0/edge repository root, build the plugin: + +```bash +cargo build --release -p iggy_connector_random_source +``` + ```toml type = "source" key = "random-source" enabled = true version = 1 name = "Random Source" -path = "/path/to/libiggy_connector_random_source.so" +path = "target/release/libiggy_connector_random_source" verbose = false [[streams]] @@ -34,11 +40,15 @@ payload_size = 256 | Option | Type | Default | Description | |--------|------|---------|-------------| -| `interval` | string | `"1s"` | Delay before each message generation cycle | +| `interval` | string | `"1s"` | Delay before each message generation cycle; invalid duration strings fall back to `"1s"` | | `max_count` | usize | unlimited | Stop producing after this many messages in total; omit for unlimited | -| `messages_range` | [u32, u32] | `[10, 50]` | Min (inclusive) / max (exclusive) messages per batch (random within range) | +| `messages_range` | [u32, u32] | `[10, 50]` | Min (inclusive) / max (exclusive) messages per batch, capped by the remaining `max_count`; the lower bound must be smaller | | `payload_size` | u32 | `100` | Number of random characters in the `text` field of each message | Setting `max_count = 0` stops production immediately (the connector produces zero messages). To generate messages without a limit, leave the key out. -The connector persists the total number of produced messages as its state, so `max_count` is enforced across runtime restarts. +The connector stages its count for each batch and commits it only after the runtime acknowledges delivery and saves the checkpoint. Restoring that checkpoint enforces `max_count` across restarts; missing or undecodable state starts the count at zero. After reaching the limit, polling continues but returns no messages or new checkpoint. + +A non-increasing `messages_range` returns a configuration error when message generation is attempted. The SDK logs poll errors and continues polling. Each generated JSON record contains a random UUID `id`, fixed `title = "Hello"` and `name = "World"`, and random alphanumeric `text`. `payload_size` controls only that text, not the total serialized record size. + +Create `my-stream` and `my-topic` before starting the runtime, or change the sample to use an existing destination. Configure one destination per source instance. diff --git a/content/docs/connectors/sources/source.mdx b/content/docs/connectors/sources/source.mdx index 0fd29b5de0..965afa6631 100644 --- a/content/docs/connectors/sources/source.mdx +++ b/content/docs/connectors/sources/source.mdx @@ -10,14 +10,22 @@ Source connectors are responsible for ingesting data from external sources into The source is represented by the single `Source` trait, which defines the basic interface for all source connectors. It provides methods for initializing the source, reading data from it, and closing the source. ```rust +use async_trait::async_trait; +use iggy_connector_sdk::{Error, ProducedMessages, source::SourceBatchResult}; + #[async_trait] pub trait Source: Send + Sync { /// Invoked when the source is initialized, allowing it to perform any necessary setup. async fn open(&mut self) -> Result<(), Error>; - /// Invoked every time a batch of messages is produced to the configured stream and topic. + /// Retrieves the next batch for the runtime to process and deliver. async fn poll(&self) -> Result; + /// Override to apply staged progress on Ack or discard it on Nack. + async fn on_batch_result(&self, _result: SourceBatchResult) -> Result<(), Error> { + Ok(()) + } + /// Invoked when the source is closed, allowing it to perform any necessary cleanup. async fn close(&mut self) -> Result<(), Error>; } @@ -25,32 +33,36 @@ pub trait Source: Send + Sync { ## Configuration -Each source connector is configured in its own separate configuration file within the connectors directory specified in the main runtime config. - -```rust -pub struct SourceConfig { - pub key: String, - pub enabled: bool, - pub version: u64, - pub name: String, - pub path: String, - pub transforms: Option, - pub streams: Vec, - pub plugin_config_format: Option, - pub plugin_config: Option, - pub verbose: bool, - pub benchmark: bool, -} -``` +With the local configuration provider, each source connector has a TOML file in the directory specified by the main runtime config. The [runtime](/docs/connectors/runtime#configuration-providers) also supports an HTTP configuration provider. + +| Field | Meaning | +|-------|---------| +| `key` | Connector key (`String`), separate from the numeric plugin instance ID. | +| `enabled` | Whether to start the connector (`bool`). | +| `version` | Configuration version (`u64`). | +| `name` | Display name (`String`). | +| `path` | Shared-library path (`String`). | +| `streams` | Stream configuration entries. | +| `transforms` | Optional transform configuration. | +| `plugin_config` | Optional custom configuration object deserialized by the plugin. | +| `plugin_config_format` | Optional default format for the HTTP API's plugin-config response. The FFI always receives JSON. | +| `verbose`, `benchmark` | Optional logging flags (`bool`). | `verbose` and `benchmark` are optional and default to `false`. `verbose` switches the connector's per-batch logging to info level, and `benchmark` emits per-batch timing events for performance measurement. -**Main runtime config (config.toml):** +**Main runtime config (connectors.toml, in the repository root):** + +Use the matching 0.9.0/edge checkout and broker from the [quick start](/docs/connectors/introduction#quick-start). Its broker must use the credentials below. Create the `connectors` directory, then save these two configuration files there and in the repository root as labeled. ```toml +[iggy] +address = "localhost:8090" +username = "iggy" +password = "iggy" + [connectors] config_type = "local" -config_dir = "path/to/connectors" +config_dir = "connectors" ``` **Source connector config (connectors/random.toml):** @@ -58,16 +70,16 @@ config_dir = "path/to/connectors" ```toml # Type of connector (sink or source) type = "source" -key = "random" # Unique source ID +key = "random" # Unique source key # Required configuration for a source connector enabled = true # Toggle source on/off version = 0 name = "Random source" # Name of the source -path = "libiggy_connector_random_source" # Path to the source connector +path = "target/release/libiggy_connector_random_source" # Path to the source connector plugin_config_format = "toml" -# Collection of the streams to which the produced messages are sent +# Destination for the produced messages [[streams]] stream = "example_stream" topic = "example_topic" @@ -81,6 +93,7 @@ linger_time = "5ms" interval = "100ms" messages_range = [1, 5] payload_size = 200 +max_count = 100 # Optional data transformation(s) to be applied before sending messages to the stream [transforms.add_fields] @@ -92,20 +105,22 @@ key = "message" value.static = "hello" ``` +Configure one `[[streams]]` destination per source instance. The runtime currently retains only the last producer when multiple entries are supplied. + ### Environment Variable Overrides -Configuration properties can be overridden using environment variables. The pattern follows: `IGGY_CONNECTORS_SOURCE_[KEY]_[PROPERTY]` +Configuration properties can be overridden using environment variables. Supported scalar fields and indexed stream entries use the pattern: `IGGY_CONNECTORS_SOURCE_[KEY]_[PROPERTY]` -For example, to override the `enabled` property for a source with ID `random`: +For example, to disable the source with key `random` at the next runtime start: ```bash -IGGY_CONNECTORS_SOURCE_RANDOM_ENABLED=false +export IGGY_CONNECTORS_SOURCE_RANDOM_ENABLED=false ``` -Top-level fields of `plugin_config` can be overridden (or injected) the same way with the `IGGY_CONNECTORS_SOURCE_[KEY]_PLUGIN_CONFIG_[FIELD]` pattern. This is the recommended way to pass credentials without writing them into the TOML file: +Top-level fields of `plugin_config` can be overridden (or injected) the same way with the `IGGY_CONNECTORS_SOURCE_[KEY]_PLUGIN_CONFIG_[FIELD]` pattern. This lets you supply credentials without writing them into the TOML file: ```bash -IGGY_CONNECTORS_SOURCE_RANDOM_PLUGIN_CONFIG_PAYLOAD_SIZE=200 +export IGGY_CONNECTORS_SOURCE_RANDOM_PLUGIN_CONFIG_PAYLOAD_SIZE=200 ``` ## Sample implementation @@ -120,12 +135,29 @@ Also, when implementing the source connector, make sure to use the `source_conne And finally, each source should have its own, custom configuration, which is passed along with the unique plugin ID and optional state via expected `new()` method. -The full, compiling implementation lives in the repository at [core/connectors/sources/random_source](https://github.com/apache/iggy/tree/master/core/connectors/sources/random_source). The snippets below mirror it. +The reference crate is [core/connectors/sources/random_source](https://github.com/apache/iggy/tree/master/core/connectors/sources/random_source). Use its `Cargo.toml` from the matching checkout, including `[lib] crate-type = ["cdylib", "lib"]` and its workspace dependency features. The Rust blocks in this sample implementation section combine into `src/lib.rs`; the trait above is an API reference and is not part of that file. -Let's start by defining the internal state and the public source connector along with its own configuration. +Start with the imports, then define the internal state and the public source connector along with its configuration. ```rust -#[derive(Debug, Serialize, Deserialize)] +use async_trait::async_trait; +use iggy_connector_sdk::{ + ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, + source::SourceBatchResult, source_connector, +}; +use rand::{ + RngExt, + distr::{Alphanumeric, Uniform}, +}; +use serde::{Deserialize, Serialize}; +use std::{str::FromStr, time::Duration}; +use tokio::{sync::Mutex, time::sleep}; +use tracing::{error, info}; +use uuid::Uuid; +``` + +```rust +#[derive(Debug, Clone, Serialize, Deserialize)] struct State { messages_produced: usize, } @@ -140,6 +172,7 @@ pub struct RandomSource { messages_range: (u32, u32), payload_size: u32, state: Mutex, + pending_state: Mutex>, } ``` @@ -153,7 +186,7 @@ pub struct RandomSourceConfig { } ``` -At this point, we can expose the required `new()` method, which will be used by the runtime to create a new instance of the source connector. The `id` is assigned by the runtime, and represents the unique identifier of the source connector. The `state` is an optional connector state (e.g. persisted in the local file), which will be provided by the runtime, given that the connector has persisted its own state before the runtime was restarted. The `ConnectorState::deserialize()` helper decodes the MessagePack bytes back into our `State` struct, falling back to a fresh state when nothing was persisted or decoding fails. +At this point, we can expose the required `new()` method, which will be used by the runtime to create a new instance of the source connector. The `id` is assigned by the runtime, and represents the unique identifier of the source connector. The `state` is an optional connector state (e.g. persisted by the file or HTTP state backend), which will be provided by the runtime, given that the connector has persisted its own state before the runtime was restarted. The `ConnectorState::deserialize()` helper decodes the MessagePack bytes back into our `State` struct, falling back to a fresh state when nothing was persisted or decoding fails. ```rust const CONNECTOR_NAME: &str = "Random source"; @@ -183,6 +216,7 @@ impl RandomSource { state: Mutex::new(restored_state.unwrap_or(State { messages_produced: 0, })), + pending_state: Mutex::new(None), } } } @@ -194,7 +228,7 @@ We can invoke the expected macro to expose the FFI interface and allow the conne source_connector!(RandomSource); ``` -At a bare minimum, we need to add the following dependencies to the `Cargo.toml` file to compile the plugin at all: +The reference manifest supplies these dependencies: - async-trait - dashmap @@ -203,7 +237,7 @@ At a bare minimum, we need to add the following dependencies to the `Cargo.toml` - tokio - tracing -This example also uses `humantime`, `rand`, `rmp-serde`, `simd-json`, and `uuid`. +The implementation also uses `humantime`, `rand`, `simd-json`, and `uuid`. The reference crate uses `rmp-serde` directly in its tests; the production state helpers come from the SDK. Before we make use of the `Source` trait, let's define the internal payload of the message that will be produced (e.g. as if it was pulled from some external database or so). @@ -217,7 +251,7 @@ struct Record { } ``` -`serialize_state()` encodes the state to MessagePack via `ConnectorState::serialize()`. `generate_messages()` builds a random number of messages (within `messages_range`), each carrying a JSON-serialized `Record` with `payload_size` random characters of text. +`serialize_state()` encodes the state to MessagePack via `ConnectorState::serialize()`. `generate_messages()` samples the half-open `messages_range` (lower bound included, upper bound excluded), capped by the remaining `max_count`. Each message carries a JSON-serialized `Record` with `payload_size` random alphanumeric characters of text. A non-increasing range returns `Error::InvalidConfigValue` from `poll()` without advancing state. ```rust impl RandomSource { @@ -225,11 +259,14 @@ impl RandomSource { ConnectorState::serialize(state, CONNECTOR_NAME, self.id) } - fn generate_messages(&self) -> Vec { + fn generate_messages(&self, remaining: Option) -> Result, Error> { let mut messages = Vec::new(); let mut rng = rand::rng(); + let distribution = Uniform::new(self.messages_range.0, self.messages_range.1) + .map_err(|error| Error::InvalidConfigValue(format!("messages_range: {error}")))?; + let messages_count = rng.sample(distribution) as usize; let messages_count = - rng.sample(Uniform::new(self.messages_range.0, self.messages_range.1).unwrap()); + remaining.map_or(messages_count, |remaining| messages_count.min(remaining)); for _ in 0..messages_count { let record = Record { id: Uuid::new_v4(), @@ -255,7 +292,7 @@ impl RandomSource { }; messages.push(message); } - messages + Ok(messages) } fn generate_random_text(&self) -> String { @@ -268,7 +305,9 @@ impl RandomSource { } ``` -Now, let's implement the `Source` trait for our `RandomSource` struct. Each `poll()` waits for the configured `interval` to mimic a real-world external source, stops producing once `max_count` messages have been generated (if it's set), and returns the updated state along with the `ProducedMessages`. The runtime persists that state and hands it back to `new()` after a restart. +Now implement `Source` for `RandomSource`. Each `poll()` waits for `interval`, builds a batch without exceeding `max_count`, and stages a candidate checkpoint. The runtime sends the batch, saves its optional checkpoint, and calls `on_batch_result()`. Only `Ack` commits the in-memory count; `Nack` discards the candidate. After restart, `new()` receives the last persisted checkpoint. Once the limit is reached, polls return empty messages and no new state. + +The SDK permits only one batch in flight. Returning an error from `on_batch_result()` stops polling; returning an error from `poll()` logs the error and continues the polling loop. A source that tracks a cursor or performs destructive upstream work must implement the callback instead of relying on its default no-op. ```rust #[async_trait] @@ -288,9 +327,9 @@ impl Source for RandomSource { async fn poll(&self) -> Result { sleep(self.interval).await; - let mut state = self.state.lock().await; + let messages_produced = self.state.lock().await.messages_produced; if let Some(max_count) = self.max_count - && state.messages_produced >= max_count + && messages_produced >= max_count { info!( "Reached max number of {max_count} messages for {CONNECTOR_NAME} connector with ID: {}", @@ -299,28 +338,45 @@ impl Source for RandomSource { return Ok(ProducedMessages { schema: Schema::Json, messages: vec![], - state: self.serialize_state(&state), + state: None, }); } - let messages = self.generate_messages(); - state.messages_produced += messages.len(); + let remaining = self + .max_count + .map(|max_count| max_count.saturating_sub(messages_produced)); + let messages = self.generate_messages(remaining)?; + let candidate_state = State { + messages_produced: messages_produced + messages.len(), + }; + let persisted_state = self.serialize_state(&candidate_state).ok_or_else(|| { + Error::Serialization("failed to serialize random source state".to_string()) + })?; + *self.pending_state.lock().await = Some(candidate_state.clone()); info!( "{CONNECTOR_NAME} connector with ID: {} generated {} messages. Total produced: {}", self.id, messages.len(), - state.messages_produced + candidate_state.messages_produced ); - let persisted_state = self.serialize_state(&state); - Ok(ProducedMessages { schema: Schema::Json, messages, - state: persisted_state, + state: Some(persisted_state), }) } + async fn on_batch_result(&self, result: SourceBatchResult) -> Result<(), Error> { + let candidate_state = self.pending_state.lock().await.take(); + if result == SourceBatchResult::Ack + && let Some(candidate_state) = candidate_state + { + *self.state.lock().await = candidate_state; + } + Ok(()) + } + async fn close(&mut self) -> Result<(), Error> { let state = self.state.lock().await; info!( @@ -332,20 +388,29 @@ impl Source for RandomSource { } ``` -As you can see, the `ProducedMessage` can be customized to fit your needs, as all the fields will be directly mapped to the existing Iggy message struct. +The runtime forwards the encoded payload, optional message ID and headers. Although `ProducedMessage` also accepts `checksum`, `timestamp` and `origin_timestamp`, the current forwarding path does not copy those three fields into the Iggy message. It's also important to note, that the supported format(s) might vary depending on the connector implementation. For example, you might use `JSON` as the payload format, which can be then easily parsed and processed by downstream components such as data transforms, but at the same time, you could support the other formats and let the user decide which one to use. -While the final schema of messages (that will be appended to the Iggy stream), can be controlled with the built-in configuration (the particular `StreamEncoder` will be used), keep in mind, that it might be sometimes difficult/impossible e.g. to transform one format to another e.g. JSON to SBE or so, and in such a case, the produced messages will be ignored. +The destination `schema` selects the `StreamEncoder`. A decode, transform or encoding error rejects the entire source batch, leaves its checkpoint uncommitted and produces `Nack`; the runtime logs and counts the error. An intentional filter result (`Ok(None)`) may drop selected messages without rejecting the batch. + +Build the matching plugin, runtime and CLI from the repository root: -Eventually, compile the source code and create a separate connector configuration file in the connectors directory (as specified in the main runtime `config.toml`). Make sure that `path` points to the existing plugin. +```bash +cargo build --release -p iggy_connector_random_source -p iggy-connectors -p iggy-cli +``` -And before starting the runtime, do not forget to create the specified stream and topic e.g. via Iggy CLI. +On an empty broker, create the destination using the credentials from the quick start: ```bash -iggy --username iggy --password iggy stream create example_stream +./target/release/iggy --username iggy --password iggy stream create example_stream +./target/release/iggy --username iggy --password iggy topic create example_stream example_topic 1 none 1d +``` + +After saving the configuration files above, start the runtime. This explicitly enables the source if you tried the optional disabling override: -iggy --username iggy --password iggy topic create example_stream example_topic 1 none 1d +```bash +IGGY_CONNECTORS_SOURCE_RANDOM_ENABLED=true IGGY_CONNECTORS_CONFIG_PATH=connectors.toml cargo run --release --bin iggy-connectors ``` And that's all, enjoy using the source connector! diff --git a/content/docs/connectors/transforms.mdx b/content/docs/connectors/transforms.mdx index ed3a25714a..b266ee1c10 100644 --- a/content/docs/connectors/transforms.mdx +++ b/content/docs/connectors/transforms.mdx @@ -5,7 +5,7 @@ description: "Mutate, filter and convert messages as they pass through the conne Transforms mutate, filter, or convert messages as they flow through the connector runtime. They run inside the runtime process, not inside the plugins. For source connectors they're applied after the plugin's messages are decoded and before producing to Iggy. For sink connectors, after consuming from Iggy and before handing the messages to the plugin. -A transform receives each decoded message and returns either the (possibly modified) message, or nothing - in which case the message is dropped from the batch. +A transform receives each decoded message and returns either the (possibly modified) message, or nothing - in which case the message is dropped from the batch. A transform can also return an error. A source rejects the whole batch and its checkpoint on a transform error; a sink drops the affected message and continues processing the batch. ## Configuration @@ -29,11 +29,11 @@ Two rules apply to every transform: - The `enabled` key is required. A transform section without it fails to load, and a transform with `enabled = false` is skipped. - The order of the sections in the file doesn't determine the execution order, so don't rely on one transform seeing the output of another. -The field-level transforms (`add_fields`, `delete_fields`, `filter_fields`, `update_fields`, `unwrap_envelope`) operate on JSON payloads only. Messages with any other payload format pass through them unchanged. +The field-level transforms (`add_fields`, `delete_fields`, `filter_fields`, `update_fields`, `unwrap_envelope`) operate on JSON objects only. Other JSON values and other payload formats pass through them unchanged. ## add_fields -Adds new fields to the message payload. Each field has a `key` and a `value`, which is either static or computed at runtime: +Adds top-level fields to the message payload, replacing any existing value with the same key. Each field has a `key` and a `value`, which is either static or computed at runtime: ```toml [transforms.add_fields] @@ -62,7 +62,7 @@ The available computed values are: ## delete_fields -Removes the listed fields from the message payload: +Removes the listed top-level fields from the message payload: ```toml [transforms.delete_fields] @@ -112,6 +112,8 @@ value_pattern = "is_not_null" - **`patterns`**: a list of patterns. A field matches when any pattern matches. Within one pattern, `key_pattern` and `value_pattern` must both match when both are set (an omitted one matches everything). - **`include_matching`**: `true` (default) keeps the matching fields and drops the rest. `false` drops the matching fields and keeps the rest. +An empty `keep_fields` list together with an empty `patterns` list leaves the payload unchanged. Invalid regular expressions fail configuration loading. Numeric `between` comparisons include both endpoints. + `key_pattern` variants: `exact`, `starts_with`, `ends_with`, `contains`, `regex` - each takes a string, e.g. `key_pattern = { contains = "name" }`. `value_pattern` variants: `equals` (any JSON value), `contains` (substring), `regex`, `greater_than`, `less_than`, `between` (e.g. `{ between = [1.0, 10.0] }`), and the parameterless checks `is_null`, `is_not_null`, `is_string`, `is_number`, `is_boolean`, `is_object`, `is_array` (written as plain strings, e.g. `value_pattern = "is_number"`). @@ -130,15 +132,13 @@ field = "data" ## proto_convert -Converts messages between Protocol Buffers and other formats. See the **[SDK documentation](/docs/connectors/sdk)** for the schema loading details: +Converts messages between Protocol Buffers and other formats. This example exposes the `type_url` and base64 `value` of an Any envelope received as raw bytes. See the **[SDK documentation](/docs/connectors/sdk)** for a complete runtime configuration and the schema loading details: ```toml [transforms.proto_convert] enabled = true source_format = "proto" target_format = "json" -schema_path = "schemas/message.proto" -message_type = "com.example.Message" include_paths = ["."] preserve_unknown_fields = false @@ -150,19 +150,21 @@ type_url_prefix = "type.googleapis.com" strict_mode = false ``` -Optional keys: `schema_path`, `message_type`, `field_mappings` (rename fields during conversion, e.g. `field_mappings = { "old" = "new" }`), and `descriptor_set` (pre-compiled descriptor bytes instead of a `.proto` file). +Optional keys: `schema_path`, `message_type`, `field_mappings` (rename JSON input fields, e.g. `field_mappings = { "old" = "new" }`), `descriptor_set` (pre-compiled descriptor bytes instead of a `.proto` file), and the inactive `schema_registry_url`. + +A loaded descriptor is used for JSON-to-protobuf encoding. Without one, that direction produces JSON text in `Payload::Proto`. Protobuf-to-JSON conversion does not decode a custom message descriptor: it parses text as JSON, exposes an Any envelope, or returns metadata containing base64 raw data. Protobuf-to-Avro/FlatBuffers conversion only rewraps the bytes. + +`preserve_unknown_fields`, `validate_messages`, `type_url_prefix`, and `strict_mode` are accepted but have no effect. `pretty_json` affects JSON text output, and `include_metadata` adds fields on the supported protobuf-to-JSON paths. ## flat_buffer_convert -Converts messages between FlatBuffers and other formats: +Converts JSON to the SDK's generic FlatBuffer representation, or exposes FlatBuffer bytes as metadata, base64 text, or raw bytes. It does not interpret an external `.fbs` schema. This example can be added to a JSON-producing source with `streams.schema = "flat_buffer"`: ```toml [transforms.flat_buffer_convert] enabled = true -source_format = "flat_buffer" -target_format = "json" -schema_path = "schemas/message.fbs" -root_table_name = "Message" +source_format = "json" +target_format = "flat_buffer" include_paths = ["."] preserve_unknown_fields = false @@ -174,18 +176,29 @@ buffer_size_hint = 1024 strict_mode = false ``` -Optional keys: `schema_path`, `root_table_name`, `field_mappings`. +Optional keys: `schema_path`, `root_table_name`, `field_mappings`. JSON encoding uses a generic key/value representation, so its output is not a buffer for an arbitrary generated table type. FlatBuffer-to-JSON returns buffer metadata and base64 bytes, not decoded table fields. + +`schema_path`, `include_paths`, and `preserve_unknown_fields` have no effect. `root_table_name` only labels decoding metadata. `verify_buffers` checks the minimum buffer length (four bytes), not a schema-aware verifier; `buffer_size_hint` sets the initial encoder capacity. `pretty_json`, `include_metadata`, and `strict_mode` have no effect. + +The input payload variant must match `source_format`. The default `flat_buffer` stream decoder already returns JSON metadata, so it does not supply `Payload::FlatBuffer` to a following reverse transform. ## avro_convert -Converts messages between Avro and other formats. The schema can be given inline (`schema_json`) or as a file (`schema_path`): +Converts JSON to a raw Avro datum and Avro data to JSON. The schema can be given inline (`schema_json`) or as a file (`schema_path`); the inline schema takes precedence. This example can be added to the Random source with `streams.schema = "avro"`: ```toml [transforms.avro_convert] enabled = true -source_format = "avro" -target_format = "json" -schema_path = "schemas/message.avsc" +source_format = "json" +target_format = "avro" +schema_json = '''{ + "type": "record", + "name": "Message", + "fields": [ + { "name": "title", "type": "string" }, + { "name": "name", "type": "string" } + ] +}''' [transforms.avro_convert.conversion_options] pretty_json = false @@ -193,7 +206,11 @@ include_metadata = false strict_mode = false ``` -Optional keys: `schema_path`, `schema_json`, `field_mappings`. +Optional keys: `schema_path`, `schema_json`, `field_mappings`. JSON encoding and Avro-to-JSON decoding require a schema; input values must match its field types. The wire payload is one raw Avro datum, not an Avro object container file. Decoding rejects trailing bytes. + +`field_mappings` renames input JSON fields or fields decoded from Avro. Text input is not field-mapped. Text-to-Avro parses JSON before encoding; Raw-to-Avro rewraps existing bytes without validating them. Avro-to-Text emits base64 and Avro-to-Raw preserves the bytes. All three `conversion_options` keys are accepted but have no effect. Unsupported conversion pairs pass their input through unchanged. + +To consume raw Avro records in a sink, configure `schema = "avro"` and `avro_schema_json` or `avro_schema_path` in its `[[streams]]` entry. The runtime decodes the datum into JSON before running transforms. A following `avro_convert` with `source_format = "avro"` would therefore receive the wrong payload variant. Unlike the field-level transforms, the three format-conversion transforms define no per-key defaults: every key shown in their examples (except the ones listed as optional) must be present, or the connector configuration fails to load. The `source_format` and `target_format` keys accept any schema value: `json`, `raw`, `text`, `proto`, `flat_buffer`, `avro`. The FlatBuffers transform additionally restricts the pair: only JSON to/from FlatBuffers, FlatBuffers to Text or Raw, and identity conversions are accepted. Any other combination fails the configuration load. diff --git a/src/components/architecture-diagrams.tsx b/src/components/architecture-diagrams.tsx index 0284943136..956e3526bf 100644 --- a/src/components/architecture-diagrams.tsx +++ b/src/components/architecture-diagrams.tsx @@ -919,11 +919,11 @@ export function ConnectorPipeline() { return (

Source Flow (Ingest)

-

External systems push data into Iggy streams via source plugins

+

Source plugins fetch or generate data for Iggy streams

- External System + Data source
{sources.map((s) => ( @@ -937,10 +937,10 @@ export function ConnectorPipeline() {
- Source Plugin - poll() via FFI/postcard + Connector runtime + Source plugin: poll()
- Transforms: + Runtime transforms: {transforms.map((t, i) => ( {t} @@ -948,6 +948,7 @@ export function ConnectorPipeline() { ))}
+ Encode and send to Iggy
@@ -959,7 +960,7 @@ export function ConnectorPipeline() {

Sink Flow (Egress)

-

Iggy streams forward data to external systems via sink plugins

+

Iggy messages are polled, transformed and passed to sink plugins

@@ -970,10 +971,10 @@ export function ConnectorPipeline() {
- Sink Plugin - consume() via FFI/postcard + Connector runtime + Poll and decode from Iggy
- Transforms: + Runtime transforms: {transforms.map((t, i) => ( {t} @@ -981,6 +982,7 @@ export function ConnectorPipeline() { ))}
+ Sink plugin: consume()
From e355b6628ac3171939327b146116865d374014a0 Mon Sep 17 00:00:00 2001 From: hubcio Date: Sat, 12 Sep 2026 02:37:17 +0200 Subject: [PATCH 11/13] fix(docs): qualify FAQ behavior and guarantees Correct transport choices, resource requirements, rebalancing and processing guarantees. Scope historical benchmarks and distinguish connector telemetry from the broker export limitations. --- content/docs/faq/faq.mdx | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/content/docs/faq/faq.mdx b/content/docs/faq/faq.mdx index 72f6b2bb2f..17caaf6484 100644 --- a/content/docs/faq/faq.mdx +++ b/content/docs/faq/faq.mdx @@ -5,7 +5,7 @@ description: "Common questions about Apache Iggy, including how it compares to K ## Q: What is the difference between Iggy and traditional message brokers like Kafka? -Iggy is a persistent message streaming platform that stores messages in an append-only log format, similar to Kafka. However, Iggy is designed for high performance and low latency using `io_uring` and a thread-per-core architecture. Iggy doesn't use kafka protocol, instead it has its own binary protocol optimized for speed and efficiency, which means that clients need to use our native client libraries. +Iggy is a persistent message streaming platform that stores messages in an append-only log format, similar to Kafka. However, Iggy is designed for high performance and low latency using `io_uring` and a thread-per-core architecture. Iggy uses its own binary protocol rather than the Kafka protocol. Clients can use an Iggy SDK, implement the documented binary protocol, or use the HTTP API. ## Q: Are there plans to support Kafka protocol in Iggy? @@ -13,31 +13,33 @@ Currently, Iggy does not support the Kafka protocol. Our focus is on providing a ## Q: What transport protocol should I use? -For maximum throughput and lowest latency, use **TCP**. If you need built-in encryption without configuring TLS separately, **QUIC** is a good choice. **WebSocket** works well for browser-based clients. **HTTP** is the most accessible but has the highest overhead due to JSON serialization and stateless connections (consumer groups can be created, inspected, and deleted, but there is no join/leave membership over HTTP). +For maximum throughput and lowest latency, use **TCP**. If you need built-in encryption without configuring TLS separately, **QUIC** is a good choice. **WebSocket** works well for browser-based clients. **HTTP** is the most accessible but has the highest overhead due to JSON serialization. HTTP connections can be reused. Consumer groups can be created, inspected, and deleted, but there is no join/leave membership over HTTP. ## Q: What are the system requirements? -Iggy is a single binary with no external dependencies. On Linux, it uses `io_uring` for maximum performance, which requires kernel 5.19+. The server starts with around 20 MB of RAM (plus the configured memory pool, default 4 GiB). For Docker, you need to set `SYS_NICE` capability, disable seccomp, and set unlimited memlock. +Iggy runs as a single server process without an external coordination service; native library requirements depend on the build. The provided Linux images include hwloc and udev libraries. The Linux shard runtime requires `io_uring` flags introduced in kernel 5.19. The server starts with around 20 MB of RAM; the default 4 GiB memory-pool setting is a budget, with buffers allocated on demand and fallback allocations outside the pool. + +Docker must allow the required `io_uring` calls and provide enough locked-memory allowance. The supplied Compose configuration uses `SYS_NICE`, `seccomp:unconfined` and unlimited memlock; those settings are not universal requirements for every host. See [Docker](/docs/server/docker) and [configuration](/docs/server/configuration). ## Q: How does consumer group rebalancing work? -When consumers join or leave a consumer group, the server triggers cooperative partition rebalancing. Partitions are redistributed among active members. During rebalancing, there is a pending revocation phase (configurable timeout, default 30s) to ensure in-progress message processing can complete before partitions move to new owners. +When consumers join or leave a consumer group, the server triggers cooperative partition rebalancing. Partitions are redistributed among active members. During rebalancing, a pending revocation phase fences new polls from the previous owner. Its configurable timeout defaults to 30s, after which the server can force a transfer. This coordinates polling ownership; it does not prove that application processing has finished. ## Q: Does Iggy support exactly-once delivery? -Iggy supports **at-most-once** (with auto-commit) and **at-least-once** (without auto-commit, manual offset management). **Exactly-once** semantics can be achieved at the application level, e.g. by attaching unique IDs to messages and deduplicating on the consumer side. +Committing an offset before application processing, including poll-time auto-commit, can provide **at-most-once** processing: a crash can lose unprocessed messages. Committing only after successful processing supports **at-least-once** processing, with duplicates possible after a crash. Recovery also depends on retention and the configured message and consumer-offset durability. Iggy does not provide an atomic transaction between a consumer offset and an external side effect. **Exactly-once** application effects require an idempotent operation or an atomic deduplication-and-effect transaction; unique message IDs alone do not provide that guarantee. ## Q: How do I secure my Iggy deployment? -Iggy supports TLS on all transport protocols, Argon2id password hashing, granular per-stream/per-topic permissions, Personal Access Tokens for programmatic access, and optional AES-256-GCM encryption at rest. For the HTTP API, JWT tokens are used for session management. See the [Security](/docs/server/security) documentation for details. +Iggy supports TLS on all transport protocols, Argon2id password hashing, granular per-stream/per-topic permissions, Personal Access Tokens for programmatic access, and optional AES-256-GCM message-payload encryption. For the HTTP API, JWT tokens are used for session management. See the [Security](/docs/server/security) documentation for details. ## Q: Can I use Iggy with my existing tooling? -Iggy provides a [Model Context Protocol (MCP)](/docs/ai/mcp) server with 40+ tools for LLM integration, [connectors](/docs/connectors/introduction) for piping data to/from external systems (e.g. PostgreSQL, MongoDB, Elasticsearch, ClickHouse, InfluxDB, S3, Delta Lake, Apache Iceberg, Quickwit), Prometheus metrics, and OpenTelemetry traces/logs. The HTTP API works with any REST client. +Iggy provides a [Model Context Protocol (MCP)](/docs/ai/mcp) server with 40+ tools for LLM integration, [connectors](/docs/connectors/introduction) for piping data to/from external systems (e.g. PostgreSQL, MongoDB, Elasticsearch, ClickHouse, InfluxDB, S3, Delta Lake, Apache Iceberg, Quickwit), Prometheus metrics, and OpenTelemetry traces/logs in the connectors runtime. Broker OpenTelemetry export has runtime limitations described in [configuration](/docs/server/configuration#telemetry). The HTTP API works with REST clients. ## Q: What happened to the Tokio-based runtime? -Iggy migrated from Tokio to a thread-per-core architecture with `compio` (which uses `io_uring` on Linux) starting with version 0.6.0. The migration delivered major performance improvements, including up to 92% better P9999 tail latency, and an 18% throughput improvement when fsync is enabled. You can read the full story in the [thread-per-core io_uring blog post](https://iggy.apache.org/blogs/2026/02/27/thread-per-core-io_uring/). +Iggy migrated from Tokio to a thread-per-core architecture with `compio` (which uses `io_uring` on Linux) starting with version 0.6.0. The published v0.5.0 versus v0.7.0 comparison reports 92% lower P9999 latency for its 16-producer, 16-stream, 40-million-message workload, and 18% higher throughput for the corresponding fsync workload. These are historical workload results, not a guarantee for 0.9.0. You can read the full story in the [thread-per-core io_uring blog post](https://iggy.apache.org/blogs/2026/02/27/thread-per-core-io_uring/). ## Q: Is clustering/replication available? From 934515fc37067a493c50d779d2c3011fd9f1e717 Mon Sep 17 00:00:00 2001 From: hubcio Date: Sat, 12 Sep 2026 02:37:25 +0200 Subject: [PATCH 12/13] fix(docs): align landing-page capabilities Correct replay limits, connector counts, UI features and telemetry. Describe the memory budget, Linux I/O and optional affinity in the rendered overview while retaining recorded benchmark blockers. --- content/docs/index.mdx | 16 ++++++++-------- src/components/architecture-diagrams.tsx | 14 +++++++------- 2 files changed, 15 insertions(+), 15 deletions(-) diff --git a/content/docs/index.mdx b/content/docs/index.mdx index c75dcc98cb..5f1bae66fc 100644 --- a/content/docs/index.mdx +++ b/content/docs/index.mdx @@ -15,7 +15,7 @@ Iggy uses a hierarchical model: **streams** contain **topics**, topics contain * ## The append-only log -Every partition is an append-only log. Messages are **immutable** once written, consumers can read from any offset, and you can replay the entire history at any time. +Every partition is an append-only log. Messages are **immutable** once written, consumers can start from a chosen offset, and retained messages can be replayed. Retention removes old data, and offsets can contain gaps. @@ -24,21 +24,21 @@ Every partition is an append-only log. Messages are **immutable** once written, | Capability | Details | |-----------|---------| | **Transport** | TCP, QUIC, WebSocket (custom binary protocol), HTTP (REST) | -| **Performance** | Thread-per-core + io_uring, zero-copy deserialization, custom 4 GiB memory pool | -| **Security** | TLS on all transports, Argon2id hashing, AES-256-GCM encryption, [granular RBAC](/docs/server/security) with per-stream/per-topic permissions | -| **Connectors** | [14 sink and 4 source plugins](/docs/connectors/introduction) including PostgreSQL, MongoDB, Elasticsearch, ClickHouse, Apache Iceberg, S3, with data transforms | +| **Performance** | Thread-per-core + io_uring on Linux, binary message views, configurable memory pool with a default 4 GiB budget | +| **Security** | TLS on all transports, Argon2id hashing, AES-256-GCM message encryption, [granular permissions](/docs/server/security) with per-stream/per-topic permissions | +| **Connectors** | [16 sink and 4 source plugins](/docs/connectors/introduction) including PostgreSQL, MongoDB, Elasticsearch, ClickHouse, Apache Iceberg, S3, with data transforms | | **AI Integration** | [MCP server](/docs/ai/mcp) with 40+ tools for LLM-driven message streaming management | -| **Management** | [Web UI](/docs/web_ui/start) dashboard (embedded or standalone), [CLI](/docs/cli/start) with shell completions, Prometheus metrics, OpenTelemetry | +| **Management** | [Web UI](/docs/web_ui/start) dashboard (embedded or standalone), [CLI](/docs/cli/start) with shell completions, Prometheus metrics; connector OpenTelemetry export ([broker export limits](/docs/server/configuration#telemetry)) | | **Clustering** | Built on [Viewstamped Replication (VSR)](/docs/clustering/vsr) consensus; single node by default, multi-node via `[cluster]` configuration | -| **Deployment** | Single binary, [Docker & Helm](/docs/server/docker), NUMA-aware CPU affinity | +| **Deployment** | Single server process, [Docker & Helm](/docs/server/docker), configurable CPU/NUMA affinity on Linux | ## Ecosystem Iggy is more than just a server. The project includes a full ecosystem of tools: -- **[Connectors Runtime](/docs/connectors/introduction)** - dynamically loaded Rust plugins for data integration: 14 sinks and 4 sources. Ingest from PostgreSQL, Elasticsearch or InfluxDB into Iggy, or forward to MongoDB, Elasticsearch, ClickHouse, Apache Iceberg, Quickwit, S3 and more. Built-in data transforms and Prometheus metrics. +- **[Connectors Runtime](/docs/connectors/introduction)** - dynamically loaded Rust plugins for data integration: 16 sinks and 4 sources. Ingest from PostgreSQL, Elasticsearch or InfluxDB into Iggy, or forward to MongoDB, Elasticsearch, ClickHouse, Apache Iceberg, Quickwit, S3 and more. Built-in data transforms and Prometheus metrics. - **[MCP Server](/docs/ai/mcp)** - Model Context Protocol server exposing 40+ tools for LLM integration. Works with Claude Desktop via stdio and HTTP transports. -- **[Web UI](/docs/web_ui/start)** - SvelteKit dashboard for stream/topic management, message browsing with JSON/string/XML decoders, user management, and real-time terminal. +- **[Web UI](/docs/web_ui/start)** - SvelteKit dashboard for stream/topic management, message browsing with JSON/string/XML decoders, and user management. - **[CLI](/docs/cli/start)** - full-featured command-line interface with named connection contexts, session-based login, and shell completions. - **[SDKs](/docs/sdk/introduction)** - client libraries for 8 languages (Rust, Python, Java, Go, Node.js, C#, C++, PHP), most with runnable examples. - **[Benchmarking](/docs/server/benchmarking)** - built-in `iggy-bench` tool with a Yew/WebAssembly dashboard for performance testing. diff --git a/src/components/architecture-diagrams.tsx b/src/components/architecture-diagrams.tsx index 956e3526bf..58b7484a07 100644 --- a/src/components/architecture-diagrams.tsx +++ b/src/components/architecture-diagrams.tsx @@ -1014,13 +1014,13 @@ export function WhyIggy() { { label: "I/O model", traditional: "epoll + blocking thread pool for disk", - iggy: "io_uring completion-based, kernel does the I/O", + iggy: "io_uring completion-based I/O on Linux", iggyColor: "#f59e0b", }, { label: "Threading", traditional: "Work-stealing across shared threads", - iggy: "Thread-per-core, CPU-pinned, NUMA-aware", + iggy: "Thread-per-core, configurable CPU/NUMA affinity", iggyColor: "#3b82f6", }, { @@ -1032,13 +1032,13 @@ export function WhyIggy() { { label: "Memory", traditional: "Heap allocations on hot path", - iggy: "Pre-allocated 4 GiB pool, 28 bucket sizes (4 KiB to 512 MiB)", + iggy: "4 GiB pool budget, on-demand buffers, 28 sizes (4 KiB to 512 MiB)", iggyColor: "#ec4899", }, { label: "Binary", traditional: "JVM + Zookeeper / KRaft + dependencies", - iggy: "Single ~20 MB binary, no dependencies", + iggy: "Single ~20 MB binary with native OS libraries", iggyColor: "#14b8a6", }, ]; @@ -1110,7 +1110,7 @@ export function DocsHero() { const stats = [ { stat: "Sub-ms latency", accent: "Tail latency under 1ms at P99" }, { stat: "Millions msgs/sec", accent: "Multi GB/s throughput on a single node" }, - { stat: "Zero-copy I/O", accent: "io_uring + vectored writes to disk" }, + { stat: "Batched disk I/O", accent: "io_uring + vectored writes on Linux" }, ]; const sources = [ @@ -1129,7 +1129,7 @@ export function DocsHero() { const links = [ { title: "Getting Started", href: "/docs/introduction/getting-started", desc: "Install, configure, send your first messages", icon: "M13 10V3L4 14h7v7l9-11h-7z" }, - { title: "Architecture", href: "/docs/introduction/architecture", desc: "Thread-per-core, io_uring, shared-nothing design", icon: "M4 5a1 1 0 011-1h14a1 1 0 011 1v2a1 1 0 01-1 1H5a1 1 0 01-1-1V5zm0 8a1 1 0 011-1h6a1 1 0 011 1v2a1 1 0 01-1 1H5a1 1 0 01-1-1v-2zm10 0a1 1 0 011-1h4a1 1 0 011 1v2a1 1 0 01-1 1h-4a1 1 0 01-1-1v-2z" }, + { title: "Architecture", href: "/docs/introduction/architecture", desc: "Thread-per-core, io_uring, partition ownership", icon: "M4 5a1 1 0 011-1h14a1 1 0 011 1v2a1 1 0 01-1 1H5a1 1 0 01-1-1V5zm0 8a1 1 0 011-1h6a1 1 0 011 1v2a1 1 0 01-1 1H5a1 1 0 01-1-1v-2zm10 0a1 1 0 011-1h4a1 1 0 011 1v2a1 1 0 01-1 1h-4a1 1 0 01-1-1v-2z" }, { title: "Connectors", href: "/docs/connectors/introduction", desc: "Source & sink plugins for data integration", icon: "M8 7h12m0 0l-4-4m4 4l-4 4m0 6H4m0 0l4 4m-4-4l4-4" }, { title: "SDKs", href: "/docs/sdk/introduction", desc: "Rust, Python, Java, Go, Node.js, C#, C++, PHP", icon: "M10 20l4-16m4 4l4 4-4 4M6 16l-4-4 4-4" }, { title: "Server Config", href: "/docs/server/configuration", desc: "Tune performance, storage, and security", icon: "M10.325 4.317c.426-1.756 2.924-1.756 3.35 0a1.724 1.724 0 002.573 1.066c1.543-.94 3.31.826 2.37 2.37a1.724 1.724 0 001.066 2.573c1.756.426 1.756 2.924 0 3.35a1.724 1.724 0 00-1.066 2.573c.94 1.543-.826 3.31-2.37 2.37a1.724 1.724 0 00-2.573 1.066c-.426 1.756-2.924 1.756-3.35 0a1.724 1.724 0 00-2.573-1.066c-1.543.94-3.31-.826-2.37-2.37a1.724 1.724 0 00-1.066-2.573c-1.756-.426-1.756-2.924 0-3.35a1.724 1.724 0 001.066-2.573c-.94-1.543.826-3.31 2.37-2.37.996.608 2.296.07 2.572-1.065z" }, @@ -1250,7 +1250,7 @@ export function DocsHero() { { label: "MCP Server", href: "/docs/ai/mcp", sub: "40+ LLM tools" }, { label: "Web UI", href: "/docs/web_ui/start", sub: "Dashboard" }, { label: "CLI", href: "/docs/cli/start", sub: "Terminal" }, - { label: "8 SDKs", href: "/docs/sdk/introduction", sub: "All languages" }, + { label: "8 SDKs", href: "/docs/sdk/introduction", sub: "Client libraries" }, ].map((t) => ( {t.label} From 19149d5a044a122dfdcc00967a74ad1a3ccd3e2e Mon Sep 17 00:00:00 2001 From: hubcio Date: Sat, 12 Sep 2026 03:35:51 +0200 Subject: [PATCH 13/13] fix(docs): explain Quickwit startup readiness --- content/docs/connectors/sinks/quickwit.mdx | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/content/docs/connectors/sinks/quickwit.mdx b/content/docs/connectors/sinks/quickwit.mdx index d572c3f9dd..9b571e00ad 100644 --- a/content/docs/connectors/sinks/quickwit.mdx +++ b/content/docs/connectors/sinks/quickwit.mdx @@ -118,7 +118,7 @@ curl --fail --get http://localhost:7280/api/v1/events/search --data-urlencode 'q | `max_retries` | `3` | Total HTTP attempts including the first; `0` and `1` both allow one attempt | | `retry_delay` | `"1s"` | Base exponential delay for HTTP retries and readiness probes | | `retry_max_delay` | `"5s"` | Cap for calculated HTTP retry delays; a valid `Retry-After` on HTTP 429 overrides it | -| `max_open_retries` | `10` | Total readiness attempts including the first; `0` and `1` both allow one attempt | +| `max_open_retries` | `10` | Total attempts per readiness check including the first; `0` and `1` both allow one attempt | | `open_retry_max_delay` | `"30s"` | Cap for calculated readiness retry delays | | `timeout` | `"30s"` | Timeout per HTTP attempt; retries and their waits can make an operation take longer | @@ -126,6 +126,8 @@ Durations require units and must be positive; invalid or zero durations prevent Readiness uses `GET /health/readyz`, retrying any failed probe. The index check uses `GET /api/v1/indexes/`; only a 404 triggers creation with `POST /api/v1/indexes`. If creation receives an error status, a successful index recheck allows startup to continue. A successful existence check does not validate the supplied mapping against the existing one. Requests retain any path prefix from `url`. +After verifying or creating the index, the sink probes the ingest endpoint with an empty body before accepting messages. This waits for the queue on Quickwit versions that return 404 while it starts. These probes submit no documents and retry HTTP 404, 429, 5xx and network failures. Each readiness check uses `max_open_retries` and `open_retry_max_delay`. + ## Payload Shapes The sink does not add Iggy IDs, offsets, timestamps or headers. It serializes the payload after runtime decoding and transforms:

P&y+b-x<=*i0dfI%9>y-&2QEwHoIt%d_IhNMcx16u?pxILdCKpdgx< z?3a`RGZxW((y3I5`R{5-BJpB)KQHLBvtfYM`_}g`$^qkCP*KqUQbyzJ#CM#2z2@gH z((P6)q+m0(uqEJHTgpOgB*M;j=^SNsuwMdNeY~RA6PP2EigrY1G%5S-#p>$%9=#3R zMM3V6ULT0t6tZ3Pjyx9D__d+HBvA2Hzc2BsNKb!sVttQ;El)1`Gr(R!SoZobQDjLlkFsE)1ijQ-yuRZF#r$f`7+hDoWF(f{G z59|#Zr@0I&`F0Dbz1i9GFJE>^6x+4#3Bq;%Q7YAK_O=d~47LI`7A0G>QwHYl$g({j z?w+ox=u3j*yOn4LklMfZL(8mi8)#SglNy68Em^#OF%^$ATX zxfLmhXly#QdaSb!9D6DQdO102W?6->&5zn{Nv_xdKqg)yAOH^hDq_cyMde=sj_9OC z9xE#=;^ocJwd3X1L|gGLhXHXzA&O6)fDorHG^J8*V>r>X5W?WbP5xV<@W3j*c1&oV z7N)JOtyP_|=V{qhNP=O+nuICbB{+yfx9crvpsQontNB-0b+M?p_a*L6MgLW%mk7aU zcX!P^Dd0nZmixB*rz&>>U$dxm@Y-NQoN3MF#K~&ej~r!wl)~VGe8v*vNx=##V=E(V zw|G=6i-{t9(f_o7WdFf_SUvBPF4yC zx_(8b5}QW1`DNM}p(Nhd7HtTmF^k1~T@d%|b!j#52m6MSIoIwjzk-{^!r>cH&e1VQ zgpQAoy>!}%{fu9G765JO=}dhlR!Gj1ybX$^-bKy-jMe|x5!w%lSvnxpAdb8N<~iIK!W$+!JM&TnnZ$*%ca~>b5e5%y+R`whYcJ; z?!Q`FF?c;<>wm$iNWHZKs_omiKhmQrrG!TK(`+0r&pkNymdFqejfDZNUst8l>S8FY zSNjgqhpn$&bXHc->$E(@sYrdm4ZPsqzLzohp8eRT3~ERX9MhKni^5Y`q&=f;GPO|I z%zs*A(Yd~LfmZ2-3FOfiv98k;I=;a;3q}a*XZyS+lq;^(jPbETSOf%9Idz2S-riu2 zn|M)VU;Ve?(6g3;Y8SfW=tf!8V*)|%5IQyUy+aZ~O0@m;_Op+3EJM~`^YU8%TCUre z{2JE)0s67Vr_z(uK zad1zzdoJ$GH=mrGibilx4s>L?IDhJlm?$wB?KrFLQ&#s8;c6=NgPD8}w~v8_%ZP+q zpJrN9WGH)P=Lm7<@{k0rxQpTi6=|4_)c8*T^kP%YKCFKK3I4_F=OkQRN5^HRXBJut zFb=|>ltofX$cJX+snZ9jAp@^?284v)J?p$1`^{q)2Z1<)Dlb#M*hu}#MJ)Q(j;a`m z{6t17PQGn7X~`%P)Ojp66UR-yB5IYla|>1*gHYPKR#@p-ML`E70X*tMe4_SE^v?sC z5>7r8#f-eI*P1K0vrb8NlvYXbGzaY}M#lLC z1lCtpo{MvW@hiXBs$vEY6RJpXZgW;+vMJZM|yU*&d zQ+o-k2-ZW|Y!po)R4Scx&%z$?Fv3&_g#Fwxs3B+U2N(OR^AHxjWhaV8JIj=B)vEba zTl>=~s-yiF|*t3>q$!|dV8-$u)V6Ov?v2>&6_vJpjTR1 zA6TueL zzh{CiaN^`H#aE(sk^=R*c!0O-rKX|Zxg)W6@bQz^>Cr{tk9!8ApkHl#UF~o3x0ZoA zDM!wKyz0pg3HM|JXl#*)=ByB{ZOxQRBm5-PT0j=%II5iCJc!Rp*&eB28-KhAyN}6a zIT$RlXc!gc6{9$8{fvzmX2}!M6_+EMf5LSTb7fjgfxpaj+67c{->cq1?K?36rKRlX zejunN#HNmv%d8I?4oRlhOXFN#yq>opm|xqx=xI#-YXlq{2h}KOK>4TFxa?9X$Fd-W z!Cf8DYz3yOwe3nB!g}KPP92#$w6Fj`Tj_j6YFb-o=1v1ntT8_ToN{tnJ06*bGg=c$ zp$At4D_qqUdrHoQ7-uCVUCh*7zy(cW>7DJ_;d+f(@7(L{C@uZcW6zn>2@@^0G8|V zPF+h+>a-hj&|sC*2_rdq)e5yzk^u579K(>K%uY;Pn42>&)NS@V`gO={ralc*fN*ee zO(*F~QIS24pb$?|BnuW@Vo-jw$zRzXSKk@URYA~ozW})CFJ zI@nGPu1x=S6jM-mFDv>3c}91nq5bn-E%Ly#GTSTUWP=81g5+kk%f4#n5(=59v{nK! zHcc}(D*BgM-r#TS_N{m)B*N?q7U z*$B1?nCk{@2ot{hHP#!uA{A8!)4 z$j7|=wXCx7^DgN;rSQPOKw{=+JeSJ}1;R3y$ua%@pbFVxlDPjqr1NZKUEU<#WI%Hmzs`^Mc{lTdSx73*8?Hr}!6 z2x#ysx5>)^Cym3vKP#!JFb5Ovg~di1+G8yrwusrh-|0o;wukg4V63f{`4Iqys?#h! z5ayWlJR)?3p|G?*+r5KT6=2K(ZwK!b5awyD{kqn=D3(RtntgGE@PA?{7xR{ugkJ-(oiP^rZF-mgUzt1$;lf#DqDa z=P%ep(?_8Xi}L_)t8cVxqEI3(gYrAnA%Nuocm$7Q8dz}jG>qvUQ@5T!1Yaw$!@ha~ znZ@Bom}(WVWwq0`(fQoZy8?d1W981vo|urG$*FMNCe2zq*;@i6_RE#cvoC=Z{ocE~ zVw*gXhAlrFPd-7O|TFsPI_(({0C zSn_)>Jv}VuxNn-1V=T-TTgoNS%&4|Iap?a+2a{Ps-7~gU+gkA+$}}}MlfBHng2^b5 z?9@PNyK!MeMeV%3uJm5v4lLznSju$Xg5J_b@|Ajze{R9dnEsxDQm4TT7iSbtF?CBh zvO#;L8s{@l^g!UKrrt638#gLe0ko1=IDgsVIA9sXrHEw+^g`>~ShJL%$Kgx1ZlEXb zA3l-#wM~UkBnd5F#Dv*sJWcD(8+MlA!GGl&QC-A%-6zytZiLKxE9=4$%B)BK_TXx+ zdvFBMR1}JqgS~k$IryM3O$r)|-d&dS<#Ir4pRT@0Ei~WS2n#FF$w*DL*%){)C)VBE z96`7TDz=__kK57Xpa|lT68~#+qLca+ce-Ora1PK)pgv{-l!3VR?Dk5>Z|Ppb~|GWMB=3 zg(=v{Kk_ZJSlosAM|xVlFB9InZtpe#|H}CcYZGB5EtAFcVdS|Yw^UDB#2UoY^_2P# zgO`Yf^6gst;_(k2%`>*eMo&#}Ky=JGnCIoaFdWL#B|F&KRPr)Wf!ZZ|dSY4TFC(cH|%j)vyz*Lm&Ax;SDQ@%O!w3KDmS zKP|-tAZ$0^ed3P%PYdYnAKo5JsAyja$m8LV`^RXqc-~UJ@uyG-HYt0qeINow9FQ3w zw6!Jl++6VvcMNVtRGg-IjyrU3aY=J>*=&5)+Ma6g#8%7yK>ZX89<{{x1v(X9y#INo zlFtr96%W>j=;aGT38zK`VAh#%1h?f};`;${!7RN`kx^_W@@gBq;Lprbl;55zC;>48 zoE%(^9O%R8)`uPXHrF!^-@W^%;YzJ2Mf2aL+rrY@T8YC{8<9KpokVQtzA*@0KR8su z&&d1?ik-)gPnIUpiX}WQVSFjQjed~k##ohCdY%7y3qf%jCwu@$4l(92-{FTQ5RNJa z0jJgC?jrOc@86#V>lieFMcN7G8Oxy=J6^o4zULQnErT!=`ICKuK4X~cJMjL1wdW2x zI*rwdlA_jliAj=e=adM54@5-lueL}qg$A>;($fXi_Je#1nDq4YT3TAr?+$z4#%43^ z``nibo-UY$Rp1YRc`oPO4<;t`S)1UN%O|I1Pl=6k#KckLp3JAj9E6F)N=iPP(%L}H z1;~NglJ#6qXW}_ssiLYD4o39vZZ5_UK`nZkv=W_dSizK-dLD| zqYq@Wx1Olr508(3!e%ejheU!6BLs*Sr0iCqTql~RUlpte2Ey4`wwDLIs<#PQ4c`1hPohZ? zjh2=T(ybw5%VcnhB^*Z+Ab*$uHR$G4fmn)($`V<8YAReWtG2MPJgO7WFo*}L1pMo| zP(Y?3413gnJtuAaTU}Qn4fEi7YFftl$XF9)qY|sBpjJ4WF{AdHw7vaFua+|PH;a`i z>xF+bxEu~Y;Io*0vxNJv$X7zxdOV2|269YuZ6-Zol!=~xx?yTszVoZisUn{r$y5yM z7d?(}Nnu_(%u))<+^fz7-+doOB%P+Ffex$Dh}pk(!Ssg;9A$v>jD)t^*>sYjoXwgq z(Qq2_;7H?#L2q26hDUN17L2y+)Er-CIX;$upv%kxni_+n>h8fW=<2O6uCySX^BFWg z{rg32Kv*?6Z?alQzoI+>w@dVMmkDsPLO-sli^F&=NQuof_W~(jDnWDZ=B-=yUiKlv zbUDM>s>VZ=WuS(rvY9X`%MhIMBjQmNOg{{I{}3uWovv`YCr=n{x6VE8E*0tb%ZjPRECCw`B4Zxl zD;z#|g4V5A6la;%a&{k&J{e_7k`SK2n>Sx=%r%fM=|Ef*q-CNttaG;qlHkJ!4~(Yc zKCO@B{p<3rjrsn+cT!MMS`~v8ba`J&hA0OjDTh0U+v1`*}~;f-!1qZl$E2qoH-91hF}_U7PwYQ61;rTNydE z;^R?;!R-oqwUnY!I`CJne8BnF_2(9Yvu3$p(%95ARcc=4ANUZ=Xm!*yza4>B)!-F_ zK`|f@@hd5%7?x~d)bgPHp4x*{TVvW5^d9mJibV#_P@KLFkwrdci^cZ4inT?8WR}=( zFMQC${ks3gCOKjM`Cgc@#sx8>-~Vcl{)_t{Ax-m5mt9dZaYg<2EiN#{=&GfqjS)%Q z-t2bO52j z#y0{GOa&F_x$7HcW#w0|UcFZs?dYiF^YVj*{l~`-W2{)_9()gF3=RpgJ?^P1G8}p% zAn;<5S3yC0Y=8CQ6#E{|vw15DCN2^=31lj{_g}uWwR!*dchI)O1g7B}m`;!-pLech z_Ixt<`}h2!>-pCFFhmQ20Iwy#pOCNW>qCqIRF$LsO1p!$U`VVHD#*#%4jGj|hG{TW z;uAGFEUGr5KY1rAN^JtONFH08^U^%{8Q$LB4=F`R*;^Fey!i~1c}GWqTQ^;*;cR4- z85%lrNKyfvv`|L(|2K4nq1 z%b+4R$9+PB=@42l(B~|fPWY!{g^4qa({^`P?Kf2a^M$eHr{iwqTy(~Z-iyr7H5*eI zYfRP|AhLoVKaF_u*-lNcW#*bYzs zgolvQlWH*8E6HBQDk09gaQhG5O6*q_7vGZecAner{_hXG{#gQg@>G+YLA@S|R~Tm- z@Dcv8(jvtwo{p_o0N#C{r~f=;5N^Dou&AsV;bfvRlt0r=*bG8 z6oyVWayvp&>9-dd32Crz8uj|n*Fb;bsn{`XPBIS|ucI)gWEKOxjqR)B64G{VM+2Jd z|3(*zi(DAjN>Gq;?IIkP3gL>rH}e>Rc_?Q`?JX_pe||rbV2a>P;_L47c-FXWrb0y* zDDGfj5>wv$FViNNWk6gvyG%(-3$z*~!mZ&Np2=uVd-wkyORA_gYP~XE7R4H=zsnoX|Z53 zGf1oSWZHr%dD#E*Gl0##3by1RM6&9|snr<_PW$iEip9AZwhKdhY)%JS zVHO%P@<+(i6dMm$>vUeZUq4MtS15i%#$%bPMHU4WjxhiGxd5opV6^_9pTLf5ezW=G zg<}Yv%ImP7P;H%Z9;^-(F`1&p6A;`2*3^Uhvwr$xg$mOy7vxXsp==iOS1kYVD=$F+ zb_$@Z`@uQ!_Wk=bKzsu#Q5&hP3*HBqr3tK_TBP$da2XwLPpiSZrKT2x3jCe=TAt07 z-UP6hLknGFhRk}gEm-j<2TT)UAcKrOFHQFIS~)$Rqt$)^5s@sJ3|SxtcXxL$M-k%S zFza+YGBh+ioq@S%mMQSwnyvK2of5;$ZV2Oxf81dqc&y?nm1Ebj!`gn;Sl0vqrg}I zQ`s*FzCp+r^^!w6&)m`y0gQ1O=|{)5EXHF4Fj3RC1Bey+{rz_E1wkfO+f&XESO`)) zJv#&19`}nE0fG@5(x;j!I;>;+U0q$$Yxf|RA|?|1x&=6xXqbcxGc$lfT8={4_J#0l zQbL{0AM;vVd}nvDo%tRy@y7akg?gQPZEYOrP zUG!zj=haV6sz6W!a&%F$yuhEh0N z;lNv=26-gS*W>Vs&4XZdy~ol|G`1!8p0U)64+g0!?Jae}(4R)6pntg5LLEao5g&S; zcKr#NbUGl-xGLm1*&{CU*~`n(&hG1@7x5!$(z5!~)m#^+;#?;fhh|ey>X?Jwtse(5y*{cU7P2cOI#apFs#aKt# zRaI4Azd{O*P5&4c)o`{7ybBU>R-a(ueu^-;N0Z%StgK8dg$~V^0Z*?2={7-rG-7M( z^*&>HyGORd%1$poK^IW)@v#7_po~Jf%+hq{^Ns(sfY^@^3_s4lJFT+VP>1ClvLo4)w#LrQEW7NyTw0*!y(KXRLfs`oW+D8RXtl$4lu7$6RwolfIWmtzAMTj8Jnnyxsol3=pZ-_M>^ zI-l8`*2Vv@H__}eTdNIH=-IxWf&kH3TX-X{8UM#HmZ8_5#IftfI`lrJ2jbMDTn4D- zV@)3r^4;Oy)+UIJgekiPbq{?oqaj|qGBiFO!F>?)M*O}9_gUdbpv%f_ydIeff$0x| zF4++NgFm}>iczZxqb>Il)m_|PG_)K}>j5f%r-wL=Bo80#ela;XFzaGMI)4qk?FNqj zr@iZpiZa`>rKq&TRzZ-c%~PVFWG#xQ(1Mbc3`&q3lq4VpN+SvajUrKk zeTY{&WeZziPKU#2W_r4~v=rdK;fzu)6x6awX}79@D{6atJ1~W6 zhS*Q`3iQ(3+uDYUx?Udu+7JWS14tx9HobFMZ{M~7b;dcwrW(lnDUYFy>H=ddH-9rr zQI!R;L=htfyF#!PRQan-ALkh&^c^|MvucmGSS!NQ19BE@WWDsz1t4Uz&;_nx7tNQs zxqL*_aPPLCmYRKEAP2~Affz+>z?W;2?sXTSDg}JwFV1Z{C-zKjz~jl=dJ-%??gcOl z#U2mK-#NX{es)QVq9-OYii`2bod>Xdpw-vC%E4V)e70`+_G#hgLZ|7cXx51_wI=m{8UrJ$iR@q@J0SQXsQ+zH~{%^$n$z@*KfIr z(Z8@DS>-e}OP~UcMx*<$^1r(y^b?%$Q@lALLuiQKSpu?G215f@K*qEnO;z$^Yi=;hfNmL1 zq7Zm;73Aa~D!oR8SmK;J$sQPCtDtiG&YgHI?MQdb(uT}O?A((Bz@mCzVN;69 z--UYxSlMq%O9Pes)%Tz3pke)65m~P?wB3xlLgTsZ?(Qy+Zkf8|qMT}oX-zB8;Q;Ao zxyv`@Zjdys6xT@l%{{i%!y__suO#0*K|WN(aOu2RCcS_&KVhMu|4e_jdM>^O{PP~! zI%;^0N78T1*MDPiJN{AMaq8SA$~+fDtmp5{6SP15DM1@qo{1P)?G>(3Lqt31%9| z1)szEnL9GZ(;ia%Fq@%UBEYTK5NC?r#kn@#wI#UP+%Qm{k6HCwToP zqcvx^BYhQ=RkZ+aB`Gte)OS)#^L{$0>Vg~qm8z+U$(49)q$z!aEj7g^{kxxS6h(^d z`y94anY%Im-XT6`d0M+NYoP;{^?D=lnU=YaZmTAo-yU5Z`Tctzc$q00(iEl_8$u3^%e~|JNm{#8O z8j_$&fwlxlicm~=t-<00bQ3Ko1<)F}+@OiGyA%*BqN1Yx z$&iAGu`vYuz2+k&Feyph3yo9&<@0(@<}7^tT-nQ(D347v(48tC+ez(9C&5NKl3n@ zF~;MSA)gf`O1V@r(NQ7gh1u^=xO3U$4F#O7cqXJG!$2#BJlKd?{spa!k1FQ}6X*#-@i3#cP z8j#%vhmnq+zU>UPCf8-N<&hfIoDclIZ}MUBRGb1`&S|>qxBA7UrS$`@1NHz&z_uE=Ar2{y;E6T z?UCa$SX%&$h-R(OjyT_*e5NpwWBYhqCY9N6VhMl#0xd0VR$_+u76^eR1j+*0k>&_{ z)~}$lee?t}8MYA==M&x}`uLEJ)8)CEWtiID#9J#&EVXN!E~VPkb=D+;MSA=*2TPHK1rp8vm^>us!c?l?dqlK*Z7m#+j`~ z9fK|hk!XSEO20|mlmf?uLqPc#tc$BgiVYeN2%xD*ciZF(_-~X%?1td{zIpSeolfdF zjrR&zkveeU91_84YXhZl#(}H{QXIX?5yxa~5O|^~FL8*azzeSWAV zV17C7s2>NWBSJTx z#pr~*1yS(=QrkKV#Bx0a1lRdQog`o`FxN}F6AZk7^MaD;y1tfES99s(t(E9!yO-0` zM7Xu}_aq-WmozjZJ&w3zf`2eCA@CHyXvSfw7!bhD(tbnw=54c!gmgj5Yr?@$--oX4 z!5d5nbp4RDyEGPSw*z#*2`l|2cWsWFvalFHth#p@ejirlvh&_&y%F7{;|N5ww=k&E zz)twx3W4{&{VQ>5DjarnJV%`d8ffJlHe#hXaS)z0}JX*kdqH!!lj zU}Ru0dPl3^>&KLLoWNBB>rfZSs)^o5%Y$rk5_E+JS@w~WlON>$lCX7`aOZ*bLF~!Y zOURZ$hD^Gbf6eM`93k$4qA;+n zIInyX11(!no&aUDEf$E3fl>fQ0i-qNLlqvL@fTG@A?+F{^hX*N79AM*kpGZp?EGUO zkUBGx_;|4P|4bV9zwN_k3=s$pMLRpLA1N}$PMC$br}Djf@IK*HM<=gzFwhFG13DFw=3IEdn}jwLjQz$pbN5z1}jwjL%jq$mXHHsEfIVijnwTL|At zNlgt82zazx(gogjhXDWBIJGEna$glL3O2A#h#8w0d+F^>CgK(Pq2O@ruBnk`-1wZW9>dRW{Fh4YYEHo(StjYxf zJD#D@IRu-~Uo-cqMJlTN`FY?3R!d4jt2^{Gc2mXvY&8f(>GY>gabLKQoREM$cw~V| zBiGMSpFW+Fl_z`amecmyNyoABQb$&WuaNGu^F3~fS%@b%h@1kl%smU#?IOH87k3AB@fKZ97fuwDVe3F3Y12% z0M;WRBa2HWsKie+`S|(*)IE&JM-F@R$DG7Za{PjOvV_|M*xziIP|=fVUKjThJ*CD% z0~Acx^Puw_TJ93Gj3+Chnrc-07CI)Kw2M@sq0l)kDwF46*DRd@h7pmK)!4pa7HR&% zESw))+o%D-tJYPFZE>b5AcH(X?)W&cVsuBrsZZYa2jPAIc}=%0a%cJdmHhfZtNDno3iq0RA<%KH~%pNx<@B(-Sz|q zzq<-v3*dL~=pE1AeGME}&NVAWWvc))<}+QPqI>!JwTX#E95lsl&lQuDddtGq|2hrq z)A%L!t4bOg0YAQx&79gXwEF}(Sz|*Z0KDnWGGR|Js3gdn6If7JCx5A`hSKl)RBNth zz5vp8YErQzKDkJm>yK&Sm1`I)1Aq`WHTL59^Gh49Snq(nIR`T1E`qyQC9&IN@N){E3l_W=87Dnbr4H>s(~#&L-dJs&W_ zAuguC)(4(p%|a`Jvq~}F2Rx8yG72ChcmaL0t2A73$&iTDy5crikgjDC4SoQCNde3} z-|GmD|HD6qi;z`)BxCdYyVQH;vr{5&8*^>7{uC%@SvfiD!7-4zvk$X_N>{p;U|R^V zmK}H76kna1nNC+P_WpDE-+9k^ctG8aW^=?7@{;2;;5Go=+{K$rP?*v!uwaj80*^w7 zm)CiJK?g`zvR}P&cSoo^`v+fw<2sJb5U<_^aY|10E-_huR%=AE$Y1ZdcMLXQ5_fh$ zxqJ>3(p32PR!Kw+V}Jf@)k~H_IECu3Z;{OZQAPz%OyoT7VDkTj4Zm>sVR!~VUha#B ztKo9E8vlnpw8Q=UU+m|-+QIS@8ahl43}V`bSAI(^iOx7R_maz^KcwzC&4S}wHrqn0<)ulQXjAx*1_d0DvU zW!ahDq0X(0?rSz~63J~|726q_tQkH&0 z^UW?FBV3&4BfrNkpUx1>)8RiM4j1BZM<5Q5#-WWkG?s{e=dB?%QecNbP`LXd<~@WQU&xme(oSzPQGB^uATE^axWoEWM6+`)2rr1X+}m};-m9eTmF zS+#qRVeamDkLxx%%T<;`n<$TZ)YM<>apG=ZG^0lGN=^M_qgKw?0;YV$+Tju*Dtmh8 zJ#Mo>J`yJkeCmyjjpUq>szN5g-dH8n+UJp>&*qusy1h#BgpcZ-VHI`=jj^JNEs=2P zSn1^^xp@+SxZgbby;%j49Gw}de4uDezx%h-6i;6V5nlf+{b1}sdDbp%!72$Y5#kv3apdQ^Igr3#CnZAjJT!(EaUvNym(lVAQN)?ai* z#yIloQj1s;gYu(=;z_EuU_JD7+!=*#KrB`HA z8Q``#aAcP<^4e-Y7Ml;71K~jW%wxcK<2cPi`L{u#0lU$+3Zh%ZayMUu@EUz(@#l)V z@<6|t31w9@yd4&13djs50W-Tt_FF4nKFis)Oq>mVaT5 zdaBIlx(#2*XNBL4^%G6l&|6p>DE)@@4X5DZ;mNpn_ikhPW^?m(^IqefP?UMo`zXvt zuep(tkzUcWphW7MqaAls{lp%at!K@YE>_IG7F=v$SFXP{*`N-n*;-~d{92oW-jyAE z%sac8c2giMzo}5HzlT2!6Dsu-LuhrTm~=AAEI%;;`j9oV`0B35-{L8ignE;{z> zF-M3_;H|A-%VcUyHdj0yD$<5BpT#|pkwmIq_;jlfu|FrGS`ue&Tqu^BWWF#jK4>NCE&iW<5p;%dH zYu)%gY?|){IrCtNs6#_Hv6(sXU9!EN=DoCew65!Vhq;^Uz;bPrBL0)lWYbm^IVdSO zHud}X$e3!0_gD{!@|5R2p6?YfGIG_jT8UU>$TpltHH+=zUmEyZnVD&3Ev1C=x!KL< z@55?nkdv?fc&4nvn`V}hpC;|+iWqr5Kvg-yaZmwZqk3`PTGqwItGX`vy3R&@JE99^ zw#(oA!XM))D=_HY68-J%*-U>XRLN${-m6-*G!wR8>c_sQc;_kIo}EE!r2(EXV}2LP zGw7<5DnoAjyC=^8W=&`!DBOAv9*J8s8tWfr(`%YYBQX{}gr5kuj4tW};Mpa4T0Jyw zuiV+e`Xnk`P@r(o5V)cMx{{KT0 zT;+xK*Ij!x7_ZwE4M$mVSz>+>q7oQ>Bgc~(DmtH2&uzJ56%b@&Gwey@%fy#p{|FF- zph_b=_PdnslVQ)qNaqywYZD;`0KU5iNLh`odHvpvFPAfRYG?hm@a4#bqWKSAOq#7N zSzG0elaa{P`i)gztD((0axP7~Mjjd+6{V|B_V5_dqW#LmC_&28yLh4g)O zE^!zr&5Ia-Egu%IARC!6&1>^j1X`_St}zrHyjh2sQ9$yb5gBBUhnx z2WHU60Q>s#Om(b3m)KmE#Lir<^?KF=Ute=IfUVDebt7c}MCvZigR)6@wV`xDSJN<1 z->Am>h5lT3vdUF7zEuWetfp7+3^p@nPE0*IQsc-kDCjVq z(-q-(c`Hh5JBOUXdFtcH3|KGFAgHaKnUN)oDBwNr{*GOLLY^&!7=9_YSu}CJE?RJPji+f$2E(AbSM5x+hg9{$zlDVZE?=cr35+%;m?^>F-h5cu;DX%;ey8{M z=kPvdrqwUdyLE)Al!g^1Wx2)?)G0Q^2fsJ>{Htj=htNUSZK~0_cRWGSC^IL^!m=hP zv2E}lt>xyk!F|*(dpcMK7eex}c{wE^?=FXF$4pzom44^XE8WQ$jokfD7kY%F!h7ND zyx1^7i>a>6v||WFR_e9N?E2f0yyF{vG-{Oeod91yoyP;XPbN`(QjO1IZzut{v$X!` z`6wUK={eD+?4K7lOz~g5ng55p^IcXL^8SAN_I+;n;{+i*CqfjBW?PaD4DZ1^h+8-1 KZ)8Z`fBttmZ0w5w literal 145001 zcmce;1yq&Y)<61C(xHeTUDDlMk`e+U-LUEI5H?6jBS@(<2uOo~1YtFfTGuGN~xT=aAHU=>U007tu^3v)6fChd=E=9Wo z{?WYK^9}ri?j*0{3II4AxBnp|GUJedEnwCf+HTs)NT_N!MZ81AN*vQ4)LP%X&_D>V=O_bis&CN-Oo!!&Z zlg*Qx&C$h@ol{Uyke!2zor{YVY{Ba4?ciqO#p>Wn15&`C{6#|=;%eq%?c`?d=sGx=5Ge%pl&Tr|6hxhp#-8^`UmC@DpLw-=H$ zcfEc6J;*bN^RJiRZ+6zdMshN7afN7jJ3&P0)gi8q?k;AKUleXT{UP$$1!Cd`F&E?F z;NoTF;9=$B5MlqXs@pOC(?Q0|O~G7DkXO)*L(qhu)r42TjFpFjkAu~OA0oiYZO&uP zD`;ZEZD#TxLchEGlZ>n>NQIY+my3^Akdud#UyxVeKOKHm{KTe!hXwLJL}&}`StRLEC25Cn`OTn{w0?G=b|z<`?r{!++FN`seri|JH!rR4{>n2 z6%ywkA(@*Axmml}LH;8mNjtZHiU`y_Av+TXOHq0+R&$7jiMyQ}z1VYyU){_p%^|MV zmJXDxl$`&|!v4?_VgGBG{};=0`b)WQ)$i9HyHz}}0$daKpEZGRLXz%oR*o)Ws@9Hf zE|hX64z|`1Pir$Dk=v&Ksst7P7yJK@SPbsWfAjbsbMZ29`fC$_&4hj_ucM2GqobYJ zV-p7t6G{ekh=V!A1>(X;Y3^tyDlI8zr|iRJFx?~5q&NxXJMygA;qiiNone0 z;$SW+bz351XX?eFM(GBzcd`SwgQy7mf0O@Tm-Ks;AVz>_!TxVd0bl-&O5kin>0Lm~ z!G2ra4iBtGlBoZrUB$TAmH9U+AF@Nh+a67JHDx+ zze~pQ1oot)7Qgio{(cuw$6MV3If4AOBV&#d|eOd$GObo6Pj2g%?b^cueSm8fxjx|)gYfKZkYe=&I-L130`CYX; zLh>~>_&fq2K?^$7CYYD&n-u*sYws-|2y}3b34w)f*1*x?6)cDH54rE$%!^vc@=8GC zK$KJS@mqhkKNTA(*ETl~A;6;m6y!!*X|c|XETZzw`royx{l&&2OdYW-ShJ!@?~toq zOE7L|o*;>K{~7PH1mkk>gc*UBax0B~gYgP}!i@a8XUpX*_Mq?M>-lmca0yF0q<5F# zbBnNihM`LF4btCj7Ms)KM1+oF+0c(j<&Y#D8k2T;n9dyMJdRUYFD=vlu;_{yy~7vr zTzG~C6H&C=S<;EbmiCGQ{VXQ}c=Bfr|A&U&_YjgRcXa)jU-XNxen7POY! zwdUH!=)2&bX&DWdUF~A;(exr%&g_!6hu=yK4&R!abD0sqx}*B2VHf^o+b}nsfd%cQ zxI@FSAUFBzH+G7tvwoB;pGD28n{L7v;nG~ze!5;+u=3ObV2WLL7xshpBLB7 zXbQza*lB*Y1RuCEzuTfh|62!u#p1M~7ktGFjK~7}&k(pL@$kOy=%B+bJ+HNbZY&9q zH3t7Q9OTC6^z=NbJ+zq;za(p_PPDZ`bDvB@gKHsP&n*Xheer321&;oE^oc&?`N64u z7P}gStkO0J%UErAMKqe*?J>h(F?z9tDY8kdpe<6x`ax#(>S^W z@AJyzSd>01!~abVc+XGbP_}cmfPJlniNbU^ue(Z$ExFye<>(=FRfY^_4ndsuw6skG zy_(ynMUZ>R%Uz7*WcNO5@vNodU5VF}o�w@YEJ)HD0FBz=XxXpGCP^cy?r19f}JL z7(OGjwB=r_3$Eo(zk9Qbe9asJj4wiuErBcX-}7y0eDe9acoEK-&We_@eLpU`syg+M z{i(4YcLU~4>JpqA14;H&U&pBF&yFx0R!Z@szRH-phUwz%^-+U>fh6_qZR#AHdkMNb zYlqMhZW=J$WNCXBuJ?Q5ae8y-3#DbD@S+eP0=Q9?`c{!P0dHWvk+Y?bK=`;)RBkAW z`@5+~?J?Y{8_Ihm{v1n{Ozv=_E}6cdfBVyo_`{psD9e~>0ljyEf7-s_#i+KNxnAg| z54xE%4Vc%37>b^b1XtIMPr!?opzivjga!#@Rn9yVC4boB&Yg8ys0kmBgXwbYF-HLR z0k^Bm2FFF*8sZzd&8AIDM7Y}Bmhb^!+_`2VNB;Me0<`$>b<1k5VX+h@S2M<}#Byud z=POm;oPC?G2j1wSANEm==Q3JdxQ>5PE(N+c{vi3G|oanP%M_1}2 zErH)OTHxd1nHOUN@a~F9giP$~oFUQrBEw@>3WyeG>{*%ae2zpst)<9NV@3R_vyGUEmPu6*{XA&f z=H~*rmdC+hbN1j*Kd1+(s3#A_J{0vAk(Qzq^a4vE;3xv{2?3_B^qfjB>PrWy1GX+J zFC)`=nF9)nVYFm`zb1CRT*FF^?QZL(pCLAU!EtRNVY713YL|xj4v}d7A58+v9cqzU z`gY(}H`Zvox3EJjX(q5(#PYc`_#3yR56Rcl0f;0V6wkDu z+@M;j-2FeMYHveU^0WX+f{#1|-c;SaDXLxc07Y6H!D(*7XZu=DEla|!^fWy=A23#i zoI-16{O@0AwZZ4w0)Eajb#K*aN%@ET*}7Nu7D($)d@mDmfbl&z`5DY0qZO)mpJV0` zB@?n$cRfIw3uIb=M0f`30S&|(!=Es9tfBeGM_4(Q5kGSN#p=h$p4*@X=3z>(pTSB8 zTj19bNbpnR64M{~KqVW1#Q>!6S6zzNTex@olN`kU7~DYSIQ0B!@JF%BtWM|w3n4%{ z5GqW80&hC@{h2|4AdTREBvRDszw3vA;;#zqSzFTkx}Um$mlXGJwNMZsMnwSGhylgi5|?vYODWNE&_A}t3!;-^aE=~ea|!>txqX3l zT(t+&Lgx6Y?pt!Ubv0RTi9%)7 ze$NUhTReISXn?BdaUzEToDRY3FP5P9uWC*F`F2GuYgV9R4+8=a14MHIt~ecoPUh=U z+XB*JfGSXJxlbbg*u;QYhBMfprw|~5c;rjfB6P6uP!ZszHh zYiE3SR~qb^Sdod`5mY&bvH)YT%mEA}CgH1@<9T;AZ7A+p&Jc7A+j#8mB1YFVr_Yah z-B`pw+IYh!Tmqjteg?v-U(?+6-jR2=r3d$?RCFfGRh>iImdJd`yIPvbTqug+Z08J{u=~lUcIKeE; z?QRPcYQ@q2!hHvqNU5PqeSN7W!z3A}o2#}1-UF@! zB4L8ubCcKp8N2Ps0T(9hk}cC)P`T|Gi=CM}SDmzouatoT#d`rcd|EO1cmH~>AHG7D z#yob;Lkwt=4%m?R-1DKI)`>Ms2R=Sig8=p2m!289E}yRPEUQ8AwY>Ze-~++o*v65m z2#&k;G4~KjA0Y@RA{WFW1Dpzg)ajNA7w{MLyK2vtZBPKtGSl80E!0y3^eesHNaZ^P z{!eF^0I7<{zODY2fl#_TXvT;EuBFK5oA)#b6YwbM?n*G6vA@AGZgSl9v+uSLwvdAo z-nj{X_18n{s&TU5voQV$c%OJ(i*up}%g_?*XKcEwm<~ zZysZ^f033R(-6cs!`yBVx}-SE>4Oakj#t6c63_YDz10I$?*$z5QIfXQ#!zvU8^xvo zi$cEK3-(A$HIi;wu4yRnPFiHNM|T7;ZZKI1799`+a$tcueAItE*&q6J)}fm<^>5&W zdM67)LymJl7=E(Tp|_lj&o}9=*ujNB7Ml%N)M6gj5+awyF2bfd07X6N84vMDQr`r^ zd-p=$>+zfL9i9gN#pG8v?=g;1BFprfKVvRio;Dl|>f8hDYDxqZ@$l*&Oqch6r<{Db zYMOv(R}0Q^X)%Sl2f(`ehl0H(sA_t$bbO49I5SZu)vaUQm|l)3p=?S)&6$W0{PMhJ zTS^i-c@9>c&F%WtTzLn*D5-A%@XSEPQMXA0WO6tT=UwV2xh-n%wX61LjWgkAB%q z7Ofwb-AmAnm8~kdN$nb*QMIJfk|FaN#8 zz&&QcHwj6C75N#D%SLAw(g9;3Rhh7ym6%GCt?D}bBTK}8?8&?1rXK8zjt#7za0C5j z_$)ccFDAY(=P3aN!ZTs>Cn-}Na;Y5dJvCJSm>?pMQe?Q1g8`^s>N<@){W(r<*@H|A z?ty{2OXF}qV=_O4D`#j+4^PfA-3TQ*)(a%@bo&)67s)WU4eKZf>qGrQyQ}12DZ7p`xqz}M>`>k z$!sfEA();%0{_^Y65qKVe)F`7RlujGEL9}@4o{D@)heUm=O@GnK}104$0H_24A+R* zp7Au@u0Q_lOjAS^x@;#4OX1^89Iy{M8@G=^ygG^DWRyS#*)N{;@X~Y;|CyjfU-D~o z0RHH^iy&PT^bURoN$k@f&3~G?&3UCaqW`BqmQ9jf1kJc>cG}+p#onvr=mG@S*3AUt=g5- zSz$96&l98`KJ{mVv+GoUE8g6(n-H1U|t zczO!YI*NYZm+~2{vPl{$PIUOiF&_E)CC9w?rLwg2r`D0dh>{0f!dX)Jmj7Y)@6ab_KERbw8%{^`~)J5^^?nKgcq zotyi4c)rZ+@+9T(reW{oBvd8;G0fkO^TpH(5uY8yOGLfQOo`ap*~SLCyDg)m_)z%F zy`!J=*q}DGhhfW%v_h%b*>_cNqyGYlpQU*kOcg(?&iD5ATE^Ddmt-~j<8RapkugG0 zbJeY*pA6F_ps%s(gRbTpheeYR5?vK~zRIJ?y0>oLI_u6&&6z?-trw;i!e!$#7Vqq{ z_-1ScHymf4zA7S^d?!K&OgZ8w44XEv9}V)g?6OF<8160Ta0Lu;7OG5cp$nfX@(~Lo zv^*}9&X{k?96eu}s#wvkbK#946A!$)df~P^fekeX6U zdwYCCHr1bfj+s=_udvgE+$FTNiB7h~1C-y@Qgg0To@_r2$j`^1Nhi7kg_7Y=2x0`a z4lSDDFR!kK#Qqp)4Z@v`Maf`m!?m5DUYlUOTHCwZak?8z6@%9g%KH1?$@oNKiJn%-W_H-)Dk#V>j;gWnpy4D z!XqV(k_*G?ik+#Mw01EB^(TA=9SAtoxs4eTxlJxTJbL*7!z)sdFeRb~!l%+V*s&#<=$f_Ee2M z=E=_aL+0jd3^HMla^Jat0H7~J9Jj%9waq};t%c5Ai*`|@O`jk&B5xlZ3` zLQuCJx7*m-vU_pWlYi#X$s{6n#0j$7Q?83Sxp#m4Dt5SGGAQMAAF93cz#Y) zGxHGl8*r}74-OK})pt_y(fv@t4f)S5{#nZF(t$7Fyv$GgP3gn4B2mu6ajkN*H85j* zLIN^Gc%oP*(vn^&`0KOqo&>mh4y?z#ErOs3g!PDY(S}KTRo-?iDNeZ=k10~m`zxQi z(_Ft`g9ee}gj`jN2^+1t?pa|={agvVim@@}%b=jUq@?;e25iYIjv4G-ZG#UeMMOvq z4Q;%c=i_o;rHs6BG&xicdD< z_eS%TTSJd+GskTZ08B#GiYT)i`a{ZuaTd+}dEebz!mX{X+6@j!pkmF|jmz?z?@@lT z#t+T#|7L=|Y*^<)@GQS6c<=#h;P9E;BjkjhVCa<@nYy~lO6U5s#oaR7v5Jr5HZSV7 z9DfeykcjDhCL-atFD3A5_Bke|gK?1M<>hHtAJ|IE$u%4sW@TkDFtN1tzb8!|e9mxx z@93!bwM4cm!_dd)%%7rP1w6^CeDhI@;eOncJn*Ahs1f(1$RF|1J@hwE9*KyG+KSo& zEZ2!oAkmh3Oo?snIX24VsY;qFR~xRu_wdZQQdd*9Jv-A2zR1hxspsbB&ur(YGcDz5 zKBj&xk*CqtJww8jSlQ(@Zd1KF&r6K`=Bs#2(RljM#hFCTu=Wk#gvfKIvoo(rfw2Eb z{#CFw4{?Y4depYC$3#TMPdA}YSU6pw(eI2oh*-BnLS(cnmW7Xg4wq>5E&Eu|#bgKs z-o%Qi=R2o9dYKS0hrTlt35igqEx=WxEvNFF4i>=S(g8H`z}z3JRS5!tNNTIHVHJ6)UUsQc_Z? zpqI!ZtjPMsoU8G2XXaIOcxS}(A`Z|!up+q|6G_OuH$J&=Yec@)I)=rHy$_n570G=s z7M&G~-TI+lZy&5R{r+l+NZ5i$w+1WJu&jF^!_%bm>+>(Yygo|ABO|)qm6cUXg$Kb;@}!-e z6{5{hacE6PI?#^=l{h)&<@^?njS%rB#T4N{H@UzUT3?L>bP*+# zU_Q#ouVNzF3e;!-z}x%W%V)Lvlv*4chUo%iN ze$jdr%UTsLe(BZ>sX0ZZ7_f?26AS_e%7tLz9yU z2Xgc3cn0R1e9guR)PR5l*mdZG{&!|09pRvtASEe@fOQ86oYWBt3IiKsV6tv>c{xWE zDJLSR0AIci$U-2Gxa`J(Z33+sY==gng6`ed;o)!%YOWMlH_Ubr#8U*&( z<>h-FU0ufWaun1|O!tVR?__0By^7p?nV;cvFGNO$fswH_!Ax$YpWlLpfdP*(|LJ3< zMANIPmAxBZI3{4Vz(!0lc!{??Q~Ov=?XE6zO-pV#K5`K<-d`$4U{u0>aBygYvX4N? z%uE9M=+d$ih8UMiVHb;E$C+QQ8T5S#h>hjfV0vy4wpNT7kn&`;#{PXYl+gG{_I;?o z_MCxK%Y1U>eD~#S$G$4VeITGyF==_W?vOV|OHE9AkfJ;@I2Yuh2gnD}1J1@5cCmUFRjDAA$xGL6ZNC;7zmLLyzrw>{O*tKKpN=DgX;-64m6n-RzQdEwCJF}Uj-rKN+-hB5>9 z4i2PTTrfg$NCIEyq<3UN;Tiu0P&wllnZ?DVbO{v`6<~Z}H}x5%%CadUE^Y$&_>m5m z6fgEQO{m$q)&iPgRY3uodMXxi^oQoqbrn?@mEy%SGG}yP^7FyoM4nGT$}v|*x)ytV zmpcMLgmnjGymEOgs1arJerl?!#@3j)xV6QE{y&FctlZc3e(-kUVm%l_Vq#(vU)6Nf zdB%%~W?XS(Fbc5DAPcKJZ~;LZB-5A5t);UZRp5|dM+(}&f_qgpYu(^SD@?R zFpj~MS}c3e9)d{-x&W#=I_Q3ZzPncyy*`Ex#CiOPDuDIJiaR{E)_M4(_iMS|rOGGANu1$fwWQ4Jl39IPT3Q6) z#f!$Q(Q_(sxaVzzlhSViMlIxwjD?@!H{@Q6H|bX7cmObUe#oA5`K=Mc>?3*>P_X|O zknqZcgoLR2`be=Ak;U`A>p$CR7~eY_HSIrL$jZ(p<*FlHg{*qPgJPYo&J*{)RbfCV@U+cT*biTq6I1np}voor?r}I6b$&Nb9E;u zOEUwE@e~VRgK;?^A=~u`xyH(b5x?1Ya9($+i>B}1hocE*+mjKWh>Q?2kH+^N%N6GEV*79j;wGlcAhc5jdaULuO} zIm|}Mq2j)*yx_*h>KJ28sPJ6y-}+Wfawq3*7F7wS?GG$0auK}ENk?RWC|Y*w>YPyj zvy5u&#T;L zU}2KF*@Yniv~iT|;uH^)*|H!`92R{kCg)oin3!tn>WGexyTp?`7|-4C&Wa-(Y3`+44VR`Ld0a{u+AW-`& zb^Xk_bfP?FzV>II3{u0Izv~y>4*6x4H z8>EuozRIknFUC(xi@dtJdggF!1GFIFiUe@x$k9-XG%_Ky*o7*wd1_-Xc14qLyzBp* zm=LrJgOkhV+)YnUN5jUJad2R(_t?e&p*OKebLq=#8ZeIWx-6-kZ8A8!xM+jva9?ds z8ddmph0sLMC}w7O47xUw-`ls#`bl!*We@g-=IUchgFfXIh! z3J*c63GS+)^{b(Z$EdT7sljm*CBr)>>Lg%Zrs~D%h4={JJ3jkBeYWnF!2QyBvA^tk zLcHtd18}zweNZMkI5;3ZQB)FP*m!HOQco!;nCE{-TgI+s!H`0WDY1dL!*XZ=j82`N zJ}xy|+DMkRGpc=vCHp2ZQL<1&Doa({;z+Nzcd)zv9e>ua6-(98v5#LgMb(!t@>?^| z%r0*bg1_|ifU}STjVYKViin7?J>n#e24m%S+SQddB?%#sNtEo9H?iO6>;}`Dyf%6& zQg}C?E2UnTMsKl*iyJlWU)Y{mU#gG_9XWzK7qmV(zNe*ApH1)`X7Tth8>r;-?#a_u zTh_melI`7c@(sm3AB1_8JBzQxH-1^q<#W%7!OkL~POZvDKF5ROzO23FP( zDYkCzl`fNoX1|w{@`_L8EG+1D8g|7rh_M7*e_-9gA$|1&0YSUM6y?BC9Cl566$S+MA39$0?XkV`>%lXZ3JjoL7O7;0`4#1vE>zQ#*~P`s(0t{ z+8S!%cx5}Fs%9AYUQh7L*RL;IKYlFJEs@O#^D$c=f$@*UkuaTTiw35)Udisk|8;Qa zrm;%awNLNaM*HwpaJjT8#v4CIMda$5Zsc6nm#QdZY~3GO9ayW=BMRj1-lw%wM}Gah zC4xnh@98EY?R{S92g=kkSuC$v@zBhvsd|*S_-mn*j0s9#@b1xJEer~}{+Le{=7$L@ z-kgi>Ufu&fA4q+NGx$MeZ;ZCibT6HE=4zkvAFtDdwS%D;$(zL0xpFejWln{W1Bm11 zk9Usov}uCRGHK*uzO3vs@bGAy&I>CY60#f-*TH;G$sf_4XnA>k=Hm9at()=ODj36> zYw&bD_ejbZ?VXgPS9w1cm0HE)(Rb)$-^J6!)f{|K8A0$2*y}$2Lr+)z| zT5mGGyov%CBD?`z#gZLSA1g36brDK)X_#AGD`!!3>`To-CHImrXmA^jt#^Os>|8NX zqL&}lr=*jVcS0sqV4KWF>~lKzwWmkT$ESXIdD;B!hgpZ(NkP`$5zt@IP*=~c+=aF0 zk;lGQ$=7E~#-R`^cHf1S1{m{%kd@sjV3A=4_-6j#;Pf4}T|Jd(V=o;VeBr4EWFZgl-f zXmBJ?k>J4tx`w&Zmx#L%mgckN$b*(ZHwT_;64xJIWK~eOe-np=(z(BjOQpKH`dF?4 zKG@y|<=zYH@sEjE?Wz-x`r^ya6Q1&V-bhtj4eRHtq-k2O*Dct{{?H=kaB|990rQA- zPg6gDBQw*{$(}Fx(dUn@yZ0Rs?AnZP@J7Tqe4L%tPyJAso6G1w`R86D&{A!+Ibr`q zapoiEpm;t^?D*3itg{f=n{`fB?@fFA`Kv+~mG=caNKVqypTB)_$no8I_))OR()v68 z)2|j&Zg59v&ZNaa{ER_WY6Sam-W_DNRRQRLzf>n2p$mhtoNO6UU@VafB|f>XwG8=g>aP&F@x1h4PD)hNndSMRaHi5 zr}UZa(y=QTX%d8yoxSYa4;XE=?TjWETqHLty|jKS~|yJwvD1MUr; zk%2Jq+WM_n#_u~@;P#ql=^Z%>SBQwVFYHO6x9OlDA1N3fTu*~AyYUFmtR28QJK479 ztK<|CK5XQ~icfT)v)dlzYQ3q4MDp1w6>i`2J0`aE!5Ks915|x&s`d?c{b|W#<#5G# zq--Y}^Y`|g<^-)W?_f9PT%E_oxjxcM%EKl}NPJ&ecvZ)1JGmYse5b$&Ce7Eno|`LqV@A5Vs|A=o~60X_lyh;FsVMCr%2z~)x}f;QM|s;I{lcr zSZo2~td*r%vF)Ap#=PCg{qOfbf@&)-|H@%wRImRnAFFtmLv4|x>C1cg_cYX(HAlV| zKIe;NuAH{}cGW|FVR5=sUmskQ-fuC=-{5Hr;_LW8gvr|v@d?k|7A2Bm$Oh}A2oQgP z5dZ{id>1}EPNvToS(Z6;KQJ%FArsn~++^uhx3PIk?t66Cui5?G2e`e3A91dhUXz;a z)nuj5q*b%j3xnD+cjk}82D`Qk1*QuH<@ODWoH=b~%{%sDCq=%>=XEpHyN$UKn+D8)O+vueTY8(Ycnj>fA+GdR~fAghY=XC`C`IIK_452TJ)m>ok zdTR9!5EVH)X#p%|_d~}yfFT%EeA(|=U#S2@ zpgI@`*zJBv1sIRk0WHf+rGx~?LI9;gsELN~xbVOD3zRyjsYSc@vho)>O1*r`Ys=)T zol;!M1;&sNKVMZw#&@4KHyyuwG}vA{LMZ?Naio^jm_<=N^QH2-*$-{$%vI*<)Tva8?*Z;VRn&%LtSHZM`G2LT#D6{?}{@VnEme>b>& z-XCKj1YAVA{agA#z(pvzg{UmBxZ!L<*`g&tnHypMpFW>cB52NwHkGr}sR5_s25m5+ z|M&Az%8~QS-vt0A+W&{I;Oc=)G3%dJw7r#x27m6mfrG2kM?_*E86a-pApnbrAFRLL z1OMu-ymI=&(`%mOjc9QDL$Y6wI}@0v7m`X8d|ZM)?2hHE>9RFu?X5L?@mlF;T~*Ap$ZbWloe; zIn$14@Nc&Y0Q8^VfVzEqvCq@%{`h3e{%+q%yfUun(R@RMTP_;LliemXGzMjqiU4ZL zm!>E(Zwyh8k#lw8=qSn5t+HryC%1)nDKIfPDbZ!VU(XF5e($l!pB3hQr|dCMw-$i` zv^`#R5p)(x6G+2ZaK#Br`~Dr7cJ?e4`==W*>50-ujlwT)dUi)D+A+ZllvtY~=1WSn z|M3y=mqr1WvQ!Jr-Gq3bGfn3D9B*vR)W(AT=3R=6FVCKeN|qTmfj3aV81AEHNkle7 zZeHGgD=I~~ML)64SV3#y=xZXZE;&Dj|3Uh8uvnDW0Huf%d2HAV6G_kdNG!)l1qJW-bx_@A=|OTNd; z19;>D?|2h#uDe~q6r4Z&+H)XXR8dL_as1 z*D@G&Drs+SZlp7s0(48fJLG8pH~M=E^!I>Z)UPZ1s`)CcV1S5%ao>1rssapCN3tHH zcEwV}>Xvj_R8_nDLq9LQ}w2XqO;Bg)3gIdkhutsD9 z^IUL7M#e9dN+B2oL+ftSClzk%>hbaMV9>n=ru(P1`4(#aSGl%rhcklgky5+ATxPJ^ z#OUaHvZZ4&@IJi996U#wteCOKA`=P*1C2Ap^T(lBVEpWIyzxBZ0c!-ULbG3Lzr_co zR4JvtM{HgCvLzED**RZHN>t=jRT)}asXmL6-@{*)f1=c~?IFHVivHgd*b@g~BL#p1 zv^w>dU)!LepiX|sjgL!VxbqdmoBCF4Qt>Uz6klH5&545$_yi#YwW0;|Y;_mwR82Me z2|GJG2h?InfJpbK?P<)oLGSXkzr5vxbrld&t^PqRkX;o~bkMSIzu*54YjZx9fs z>@>}FLS%EOZgg4R@s~bg;rLdI*CTi=^Uj-f?TbVKRtGvNxnW`T9tc%|Jp{RtS)kq!PKDjda@{j$Mr; zjEi&H9`?}q$E?6f2E!i?ich}1+a8v!UP$h)73@o?aPNV*xNM`yo4!6Q&C&~SISO7% zSNt>|+u+P>$&u6{nD}I5%3fXqoe>0Ppi>AYNioCPooDR3q8_&I%r#8cyEFTqY_&Sn zUiUT)kX2<+^k?{`!*F&W#}P8ki{cVshEtKyuRSk?=v&S1OUI;`qJrw-h-D6`ne1QMH-fyAC4I!)(}oq{s+rRq#jF=yAuYZ z(z5Uzsd-9zMZLNO>@|-D5b3~e4F=~<4RcXMoG?BR!*$Ficst^C3f|Loms5ThF&-9i zUK`QYPVQ`$8O~RiU1Z>A!10OCMQ*k&J8-%Tf(IVk-ftU8@!u) zZtWkuL@@YlABY$0OSQJiSdA1DFjc>8ZI`RM2_v}AqMcH?%ACbO&XNQqv0M==#9J-Y z(Sw%&;-)ICS?}gQrB=B2{Ns48E+CVu7^J?^R~WPaoSV=Ba0?*@9Q`K+{dTzORJ# z{64NqGG=~z;OJ`GA>Lr>Uk*oXZ1jF8_?t$lW&)Ww zyklc^At5PAsw&C;#UzIS?Hi;^k1DsN$=nZgqnQekr05R=10~07hu)CQ@Krh85bD>t zgwt>EF7_4TZ{66_tK=tiV*YSm9gzWEzI^;j#Pw}rZ*#jz?fw;}!RPy~fdq=pD=qVL zpKPuXvrNwSPH}V96N#^CHZ32prYO37w|-meGe6nfxjcC8s>Y&s#hRMBE^_(87iS9| z1a@s-byKf`%Ff^E@ zXlVWDES2@_5;b#jox1&M+Npx(CK??UR@%Xw!RniTW#rf)|NvVEwM`$(kIiy>2)&UoAuFw*6-eL!R z8bR+X2D*fYC7PGD^G_Heq_U)BIEa+(7V2LsipyTv6Y24;K7D<5le0iX*88Qd`d zrJ3jC<$Q&NDByjCmg%%(hphQ7@nuLvTzq`TLFZwe*Al62NzC^*v{x)Q&6-t}TmmjX zf&EF-@ED=TQdhhxX#%qPTLf%I^{=Lv!)O&4BEkYPDcjl(K>OYrzq|H2DaprZQVtJ! zDFgIc&Q9KyY1h5rJcONNR83zMG`yj=slF50Ns)Fk@T^@#i7An|ss}e`7-sSWB& z#hK;153*;D@sA$n3Ec=O65%Lqh4AOS1B4_TC=R{ho_#(=8u}}AIjx; zb6j4Qy^aHQNt@*cSH$Em`3HAL5^Ye=jx*=Tx@?^}gGt1vyYr%72$N#b@#&X5)=E2a zXI5m(CNB{$EUHqiM?VaM*DpYs_kyQwsT?MVe8>Lj3Ae96LBs32q+W3TEB7BS7AytMv<>Kag5nC-p#@M6KoU!d8T%pB&>0dKl>7ZlrIO#iSYr^;M z(lz$e65I7phbwH^@1xY%>B?zz6wZ*RvNDgJYUhuh%dY3l+I3rTflT$vdWOs5f1_QZ zGpSfJZ#T2#bY^Ht9*deh8tCFj?@)_5l*NFd$5)!8a5(YFY7Qq_RoQW`|#p>J`t=#ibcv2s*2G(m;W=$}=ZE7a^x$s}}Pr=BmNVtKDa z`YiC7bYyIFzd=uUGu3Qw@Sm&GDjdWh%TaSPuk4V_2D0={2K7+y!3ETc#zAn%walG6 z+%2V50KXTEFVhUINF_n~=)}3cv@De`9QwNniPGh)?=atpQz}jl>c|SIBp<4Ig6j7# z&mmY&V)pu;L$5j4>%_#6YzDiT<{&WXQ)Sx86cI}Ji0+4ax@@oaY&t28Cx{Cj;NdHg zh!@A8I9$YbqS&OhBQOd#x^+D`zyIDs7oDjVOLGl4-+2SJs4Yzu+7vm=(QNDMHm}@!H)qa=}+2Ac*PNJx(Voe)Swb&r& z4rFk*0mKiVBXYNH5Ad?t7o$wPN|u#GxuRPx3yD zyRz?Xxw3Pu<(wio){O`^ftfJweJ_z>y{5k8SH=}*W>q?jdZqUe5G4Dqw`fRlgcCn% zuEcS-lv;t_e1=VLcyCM=PfgF zknq-^J6ui@M)zfwS^~ujpBK@I-+X@UNe3G|b1oYxG?Nocu#S#9qej+3-8dHRGiU7s=;Hy8>S&W=5(T>+vzp@50ybnlEU>+ z*-K={_sN7t(KwvD6;48{&}Mf+!^^swMc43GNVvB^yAyyec%M!G6`#GX>)D>k$+mDy zjs1**l6rCF)3p8U4U}!TzX++_!iC_eGYqdlGw-wVO<-+|!Gn zCRplbjc-V(l~3DC%fA19Hr&o*JO7qY_}F;+W1ji!it?tdvzGPjr$_O$cL}`j1hq<+ z)IW{KqmYyJ5emcISxHSATyYYRe+83GwBc1N94pqY1RDCE1ilPmd{%?F)ER+KbpJ)S z2hI*WNOQ1B{#v$QUg=f7W?H_yQq-Eh+fI;qNJQ}A6=vEL%#vTLOkb*uKhB92M@G#_ z$N+>Sn{n^>6*>6VitP|rwOdp2{2ify%NgMBe>3J?DMm6;x`*P+^t>>1bUpV3|LIzt zTNN!Bl8(H~?eMx0as%Kk##btf`x{~2;tkO z?Us`gk~*gwyYz4OgY|S+9?ZWzx)V*p+gHKTYXj+uiq(d%$bzvjn8XM~x};LP2fI2T z@jcy!B8%R@Ws)FRllk`E+JP_Z|OO%f*7p6L*Yp)opSTTU5mP zQ!%F@jJo6G(vRJFY{4b%<6mKGTxc~!Dn&y88e*M}DAO0B;=&O^lZ=IJ!I*MZC=rgo zI6UNs0+=Q6ExURG)1@;xFQK2$AK&95RZr==sAWFCsC^nn6@c;Rz2lj%YFO&pmLAF( z2A4i(XkzHakIIMtK~fq}dH{_;r)I-9S=*nOOh@Nu)$;zzR>_&p7pP^Yr%|4-b!e~S zeFPZFN|x|ik@GfCB60C^bd4L+C{0%{M~gqvB-EjvN$U$#1fBt4u39hXMXICz>eZ` z(kG_A9V-j%i;`y*dW?Q^vrv(~E9U#Dm?hCBZ{S-?)L~v(GP_eYdOF}l%4OC~a6oE7 zzw{8+wqvoLrY6n$6^r9BUb1^_hdFUSGz)QOMI~<$J9}%{nLrO69v|^#+?wO$5wLm3 z$A8GtV)l$u{*;??L}m?tvl-om^|a6!X^3En_h0%uD@?DHE69ZO@$sOers3e23lS!bA#dAw{#yO{ftXhmezbUgF7~wS8PsCZ{x4OCYO3C z^+FE1uP1y7LgRKU(FAeK&azYQIJFBWT{Qk4oUTcP*`1xA%Ms#H1b-a*(4}CNDPsP6 z)8hD|dHJ(uMvqeeH!2Cx#bqlS2aOcA>HjvfKdX≤H)}fQ*U?cZOa6ip{6b7~wH7 z0YhI48$S2F`#ry-@7x9Ui)YWDzm$=o6%<4clEdYPglNkvC|EPy#K93s>AOX2&pa}s znKS&Y@109t*F^8TqVkHBM;+{9DW0|NhZdC@Z*xk<&22DszCSotIr|bM6rHhg;aji$ zG_B&qv1QR}@p!T}fvkn~=YO%{?fm_%=_Jr(_LiDYBgcAn=369QzWk=!b_1-nk-X(-Ew}o2oTrJw{{D|1KAbB-c1E*O8nj20XSj*K zc+s1$^Bh!TE#Qf=l0n}3lK2vhRUE(k*xg6Ek3z&~_o;oLuI1knL9qJ&~tzP_W ztT}1gZ6Oh6Rw)&>Ym=vb>%s_%@-lTZt^V%Dxh->@w&X23+55skk~@5NHJW~Xg0Y>iq^K=?U7;57LYH?CNLzeBV=f<90= zYjJ;N0GNZ6K8I+`YZwpP@H6Q%Ux+4k-F|y;bd>Qjv;qAXId4<(QbK~}be&hrBFkd- z8TvI0-O};$yVkzDi|uoiGcxKEPrycESvg6_B=E4kE2FE6FV|<}iq^Kh z{SY1FT3AFx=Mdu?-$4J4;S#2fPsH@kUcP+nmu&HbE2^_o%Fb;1%F|%Ya~?-_BaaKW z%%Kl%qveqVv@&e(wjU#yh=iSNwg94vrceGcP&IX@!)}skV-V4%%J;>YdNK1 zxK=Jlv~U7PxTt^hFZ(S|lb&d1wXZ6auTxCNrEBUArzOH(WXe;YyjxOqJKLdM>W)jw z;<_XHfcgpVh5kIc0IVIakOwbhVwlZedc9@BNj;hDn$%Ome8B=P44d4 znt5n^e0&b&lMjlEllD{kU0-co&b#aZ3rd$te3AC*PKZJ5v^&ZxmeRF>qla@gE!s?) zHdQ(EOF+Qf;%&Wxd*izX(|b#)ZQ!!TCnar&moY5RnU{fC5r43z{7pX7MdgH1Iag=4 zA*~!YH^V4(k?&S^m=cW8@>dA(8X2yHna4ifF~ue4S#TC&Vv0DKcByZnbZ_n%w_Ex- ziDK|?YRJl~lG+%1^8V%EySTNUD`#3%PDnYLlltIJW)CVRzpxZa6io0Z z*1OAfri6fj)J{v;gI+aau{wx%;(A1ZfYH+uu(CC4=L|$ydQVIfYwcMOQ-{skaBQvy z-`q|auPfh2ZrbX&M~(~9P>z$tlGtnQH2*q1S!M{$l!}#i+3(||s@j@0D5&Y`3JDXWeU?v=lb+|8j?0rr*2OuEv-ckUma97mu*fTfvEALD;!gdH6H3*_zq&>QTuoiwQvnJ*Aom`fu3sDc zfTMHGF9Z#Jrn=m{GOjo*b?JC#elY6|UZ9valI`x`9DuadE)Pt{?bGBVSIPfWA`_2n zOSM}PqWouS?+d5`mJ!|-=lG#mweLf5&21N>hYv{v4FwE^IhK1gW@3%LJ~mr_H(kP7 z)f+{u`A#xASMbISoY?^x!^`JvVnR;+${{@KGcIKlv`kWQeBP?XAM}m+85q8XrZmH{ zL8KD(e&WZ2t*Y~-gWJK}_XM2+bXbEQa&Y+j5;QAh*qMUo3&g>xznLY}h0Vw7L8<6y zJ-1eK^ggkz&fiuTD1nB4( z`7`5=i1k0^{xpGPX!%AxnqmQ@&wD5)@9ymOnjnuH1YBw@*pgf?h7K}~6tT(pK223S z9S=iG$69m7NBm}W>e@Ao{sJXyFzVNy17Qs&?6WenJ7%SE#ydMUYh$rBu1n6T)hnkTuAMnmQla-d3bt%v%5EQ(-tS;14WwxyeCkfQF`jNK zd3w0fR&2cE_;pw`j0!AAxmTOHg$5VJrfURYN7nD1i3aGPihGe+wb-P4_f8fGn=ZDB ziZV8app)z2bkg9UBD{3N!W-`X-k`aJr{?M&ZEk<&g?W@Ba-Ja})3g0rrZ;omAMAsD zX~vXiXPupKNcuRpv3B5y;Ehso2%mm}P=>x*ck^_aq-Q;e=a~u_L~tKe=a|^kY;2+S zQ?W1eG$&*BR-y$owj>f=3Rt`_YCGYgtM(4j6J1MqC8M&p4|2?>69opcZ-9g1QJp;)~^DR@jL&c&Drs8&qT7r^VEh(z=Zko%wF8L^& z==Equu~d&P{jntAbv?y?9mlD=(5kaOeq@vzZZ-J9HkSY~NNSs?saC)=a5@?VhQ{Ca z)zAcZYq}Tuc`!*%?YfcTyk8-yxxM$H7z;eaK$r^(HXc!IEie1@y{r3B9F~~qnM?=N z4HMB#hRCl00cuCzn{{`dQ)jidcD~3O3j6vM%OUI5@ySVOqyhD#M~z3ql}Sk_PK#IK zSj%6gh3z;*n4pMPZTN&xCdwrvY4zVS+)+X<94KcnPe{R2`a4ue1}e>#6`SuPYa~J@ zk25ph*YxcdFmdReB0&TnC?kvv!#G%Db31#NjDbrn_$d5_d4S$|Fxkm!)(NNCM99VY z>2?bxl1{VgZUSFT-SNoOf_x{n(1ioWB};jSab#h2>_@H)V^rL4N|h+NOshA$Y!4s$ zK`ob}oTYMn&h4_ZDByN}U_Y~T09#CFfK)aYS=d7L1+XIM=*O^?Nxa&np<-4kh>siz z77wF>{TZA+v9T(p>SSBURMi|BX6CMFgk|N?b%`*by|GC-<0Yf>7FHe1(v*eAUcLV> z7inLp*MbXr1@Ifr!Oo)JpDF>v3~kEUZMi3D83x!VAD`tP7E3mKzJ7iW0WuDwmYmwz z)Ahl<_d{!%|2Wu@qE=BmxG}5GOd)RMS@h%tIYE5y&;d!~JNyEYdvjhBz;OEWodL;Ykp-8vM4I0nM48i*DDY;6#72s?9@xTrg39p`7gta=(^kZt zfsAVLF47@OJ`HkyH}c9!>A=+?I;Ed5rO!Of1W#r2wb9b0+{hB7C83^c38JKBVhVs- zQNG-Y8u;LPVy(`DgC^Y_40`E!9?|-b*T9SUO=X~&(|9ZfN|>Jjb>SYl?szFL{urC8*Wc?6wpX3sMz4R20VOqv0=EZvjn z*b~;>;bK<6a3qpHGouZ{J?PA+o;-u4)cOj&cXFmRnbMLaltEq3u`B_pR}~TKbIiZWj~RK&jIm(Pn;{$nUs{ z4aPWH4i18>CpWW({5112UPZRQdqQfrd3MJGvMmO3H>?TiH#Yo%d_WR^a>SJg~AsBpFVvu@%3aQ)^k`%3k|r(-+rY2;sxP= zk%LzgSVi>p^%q|L&6Y36r+!6f)a+B*(_Q-Z%;97)8cd~xS}ts?EvTokler>MzYR;j z)0Lx8FB5!LWqt}s-!Ujw&6GUHPLx79Drcqn&&6m6fVhmcxf1wlR?rM^WnTnh33;q*rb1gp*x(}p`)Uv z-CK&=3X6|#_~z;QemMIEOiXUi9BRF$;oDAlgZRB#~sEfqqDL; z2LUr^r3($++kx-A%ac#p5=3!C{m7HsQm3u|4y+`JD+4rF;TJHe)r0}&{VACaz8G{W zBkZl3Q@?06Q01rq8>g|`+a8WC(yJj*-|aiyGafm=UYA$9Vc77=axh0N&U%>z9QLF6 zIvlURaxV0zb?mPRqepj$p6nX%y3%}vmNXf7gi^B%`82F@h;a#}3a_-Ko<{am_rK%B zzS|oImByT%bD14;qD1(VC|Zr$_8svciVT~~Fqkbw%$nU0o8h=wZj`A}Ni*O4df$zb zAIqoF6EN<186oIh57|uq-d`Qb^MTS9ruxXVgTKp3!{32Q%p|u%rIJ;0v6Q~J$H{U7 zKXhC{-n|!b-J*8+=|3dFtg>_l$GGb~P!kNSbaFMtYv%PF2$G4)y)FK5hcxgry+Zn9 zmtzZP3!VbtkGTA*EAp_(uyA&HSsX;Xg~fu#!%Q^N@n2fzziZcj{N(LDS#r<6>0^^F zTAe=oclk^@xny)$2g)YRG~G@NYA#NVL7$_X6$z{5=@sO`+DL8UAM*vUq0;#^fITdh z%aoLuMg1FefKsvS-Zuzn@=eFNz>e`!S(yk%9|*ye^E6){tPbtDWIz zhqIy%vrTqj2e({kl>_nAi;Mxe{HJJFu6pq}A2OMhdch(lm#fZObAG5sC7}Zg*!pyB z5}202vvrr>j;Z$Ilo7xb2u4_X*8w{R^jzvJ#C$nKaYOBk9mH)fW$I113JebzKv?ut zechmcK;Cs!4NJDdCe!_-Q{XOF-0RLqtk+Q~>vsIsLddYrY$o84tbQBICfz3cQ6n)p&~Tw>$8Io zN>E#F5wqydeEW_~Cgj(qkbXtBB%52-j6D`&ZYcS|PIl$83=i9{0dQxtI^lJAuabSfQ3bjyOpk>H{feXYj+kG01I5nffI0zq z5&an(==MetdMMf}vrD{1-zisY?coH=` zJC;XlD_wUIzsxV)*_yqbQ?hUe`*uYvBBqxfwW#N7dYTiGSDs$Docx{2At0csdX7SS z?(16~z@(K_(mXGc)6I+JJ|Lqe@v^G)C#ECmP=b$bEy|4TiSh8vnqv~%USnniv%+W4 zf@#GSFclhh@xjw+2#FT6IN4Pqaw+|inyL>J_~pW|GU84LYeaKLqt-6Qwg82Vs%g*% z0b$dN*!}(W#qwZs-gaMtsnBguqe1oa@KMTTtqE8wpyyDty+gVf+wd`qrG$R@Lbp1Y zEAjo=HMY1effId=e+h*sgoTAkxw3mY8h!f9tubp(Kj1F5M|KP^4JX-$#D|4#l*0mH z1&-gkxIf>A-WLY;MjOO#PVyI8&Krh>7n(aa{(U1ZpKC(0)P*F<7>G0--WAOmP17MngJah9Y zXLk%pRan%O!oD|F3ije1lPn?^o;|(O19x9Q$Zy}e^)zmnnb+no>nWatZ;uelug>GC zzKWP~TL8r#1{C;B)6>tOW&ELD{t)ydMi6p=P0kq)702x_|kI^-5JuO~#6-)e5{P&6Z22FQD9ps_pM+ zAsO%){IEU%s##&X@%xQdSBqJ#O1d!1CaGOzf>Gl8kM-I!AP0fg2o3GxSf#VI;CFU# zdSC9OwUjIBo>1+S+^vQVt(6X?Jjoom&sif z3wHEEyXvE3&+~U^9LY~Ffe(h~eadZo1>qQpyu``g4^L|R=u~r}E}!{M*q!|~6uws1 z`DY`wVk#b<7^AP;iwE2^Y!0m*@j~}JNA<6W)@%1J-^~Gc7sz;|-0bo1@fFQ?unK?I z80R{2HTiEPe3T3cOo~4h+m38OYb$j6H^rpKb+?VSd`b%hxBT{-sUTOi*_wiNv;pd3 z*eO9H!EU$y7AiWm5)+-#0t0d(mp`Q&Cwt3|N0avSNdbNP;^$99(Ded76UowMOWd3pQw>dt=e4xOMk# zt9k8tGpOc#{r%???57;xDahPxNC3^r5BfIWQhz_YhkR<-EerAP&;I4YqB;Ap>9MEV4Zx90gB?cz~S zWNkg|^^e=;JOC4#COUiCPdSBKEw=0Klztm}2eoOcWy)uPgN1OLjVb@VW!4E>Ex<7I zmfSWv@^cg_iGVe>DwoX~gitLCgjc;H*wM&e03v~qc0mi z=js;~(it7Y(qK9>XZf4;W**1_$SDDd^cJI9mDxFl)8XIGR1)^g_mZNbnyd0^uT>3T zNX2o5HwThF>|lpEKRdWGP-9E%Qrb)GFtlPr*qVgrA?SQ`@@mmYUw?aO;v|3!i3Nc> zDp8s0qx_g}p$3OCU@td1RwBvm>S&S9hDOguwFO^-g8)T^{aQm?cm@x_?UjTKAzp9! zG)d5=0oe0y@B`G%zoF`P9gm_{_*#Dz2_0PFwNVOY@2U$|oDO0TaL*5}aFnUDl$cD4 zh11}m@z`HnF_(5as<&q=&0;7I#!U46k_rVYNJ0IFdrI`7&NSb@cM#~! zkijFz#q5aAyK=wkrA~VUi$nrnv22#o-vhp9&pz@v{AIfreM z?)3FD=tj54j@bbG%x%(YG%N&x1#G(f(74a-^%y;p#3yEHp0VKrNE%3O7WJ~$Prh65 z-z5BoggL50jnF9ZzFMBlr3@r}@=ec&n1tW#N@gavczraJ%13-^KMvRCY<{TOrMg8ghul_m*CdmRVZocFQCR@k}@RtTU^Xvdt}i z&sMFDGDbESRs7T%jhb_ z<&G{k67Q_7_4e~jl|sIbH`#{B!vrXHE{^pmAp>y)sigw54*t=4b(DnEvbKg^8LQdh>bj+=vewA)`?AmM4q~$`}|ORYw=tG*xU-xc4;gKN9qciRl1n zI$oL>r0zhz-?~jA8rz>1Bj}6)D^SadmBG!!H5)#1TyctABh6fA3fv1^(Aq-5L{7}= z6fVqV^5?~}^gWl3k!%%l<*3N3m={1o$*x6((us(8dNe>}m2TnaXo)H9c*(|#9AyGa zny_G_o|%QV0VegLh=mBku&{7Py@qk#QxH4;DPDUsm2~>RFL`?*+?Isbiq3e%v*f-rh8zWNYQEh46$xxWo&oT7ZSL7W`l(#LBhlR|tt` z_yNn-S7NGGr}mH!!cq!HJHZgvnj|a?_updJrF(YFhK`P&?{@AqVYahGf4yOz8h*W8 zUeLAs_v@0DdN@nMPAnSaMb)EwvT`QGtrY$AV9-9CU>A`N87;%U7n;k7b&A-1kl#@j*J z)!=cAb`3Td<=<76yZ_(;^2^TtKA5~52nNik9;nxZRqKA-i0WU%Q-uLkI+fT7QEMty z@6a?V4rmw{{2=;(eZ!01MF9ESZu8h117{7?V8?cz+%f7EiV{)GCy9_Y#NpVk*w4XE zp-Q>eFfZqla%9~xYXL?VxlBb^$pRqcw6>2S^6Y6weYx89>DJm($(ecwQP=U_jLzW} zD*POYO10nGEOM=RU4oAAXbxOeD3vzVbhH&19^wZ)ES2lE^09WHQw_Kn6WzK zX{Tw@+Hhu*$A#-9X#=bsqIac;oX6q|^xhB;(zmkTiCVG-`^U~v>0{LRmgCm>$`3_$ zK&Mje^EP?NrNhF*E&SSXwpTn_te1tB z)w#Fw%CBPL;!{$}l$s^bsdC20Qg-c7BY4Ph)mdmcjNCppt~|{cnzJbfk$mUzwjLH? ztBlv_0WgjHRj3Q>!_CPa&#Rd4duQHX!m~p^m6s~Fi1!ZF%=1l77%mwv;Hfb03krI< z=;-_l>x`9fa0p!+X%&qcYW|E9H(F^f97Z+2Y(yg@p}KIhq`StpnRh&|KOj5C+IIfV`@m z@Sf!;Lz(cl$UemE==!=l6;T@dc$OI;Jj-$8C2&2h<>}QV&F3|x z3>eA`R*pOV(@Kjt$nFA?X?BjQcvaZM)@a&g4+jComf`kHy~mL?a7oT| zPz@n2N)uV3F5tYLE<3+%s`Nf>6!gXU2+r?larL-<`t>A>}AJR*6WHFSzmI`wXA zcx4=+Rp2pdawxc59ZfG3eFG)LcFkiu_~LBuKJ@CZg$-&+$nf!-cNZB|0L{~KJD~>f zG#$b_z-;C98HcO@qPf6ya<#6?ac|o}0JbT({Yxx!rp~i!x1BLtqcRGzw?Iz^`T{^K zUIA(atk=~#=pCWFVA84~Pw9IJRs+ZcBnErwBWh~zw(#la0T;yRvzDMvO$Shq^ZmpZ zV3!auHB^3CJxS3dk(_v99`cxzY9aB^ow=UuQ{ngqY|fFPDPkN}gYST1>0-abT=}98PXa0zW4!m zc>~zfBzNnJ@J1o42?5!NzZb3gYV_W?P2{h9KV)UyWv4{qfS+C8z@RU~9*;qpsizLj z)wRA9>I^oW1_#cs@dJnnMth2nb!Me&n~WMqzX@z(cy^uuxhcNbe#ddOt>Uc8RJ_C% zi-@l7hg_pdiOUlrfA3_`Mp(1fCrg7da8@65up4wl>GXc9`IoxvXZ;_xl!EH1tyHPl zeyNzpLDO#Ifk{oBm$t#shu)XO4zl|N)PB#PuhMG?pki2Mie}MBcled;BH)Yx%rQtC z<{*CAdy!Fj)~@pHUZYc@Loyc*nb2#9Mp*gtOAvWTdD)_Oj}y?5YA%$V(vNNo~6}2+=-|VQ;Ql+flEuiG}6lyLHB)VxA_>wp&cD?00fle^OnM zc9$jMe2bM67f=h&qPy?G+WP<>-;9PA`u)Y#p|_+@enK`!?-Br6 zZ?^y8U~B;Wg0Im(p_ikpS~k!#8S8C*H-u_>4!^S?B4>>o!M4 ztfvb-CrFniS#iFJcsbb%N*Akkg*>>L`S#ZpVbiHsAi98hj|gae%XymJL$xh7gI&zC zqW3r7_~;FF?2ph!$f%?e@$>{(63?fxdvnqqV)9-q+HmFtQoaVM!OBP}F;`X~ zbkWvxzv;0E>7I@+t0|<2JptmkbF1=bqBxklOmI_unrDYD5n0n_1!4{;)t@2{Hyq}C z$M!eI{UO4d-v{o zQuoOhs16G3x7;ozt#nwTeuRc|zb%1u&|d>C1Dj$ZNjhOn;iuva4D9u_IC+{k%d-^t z+W(YWw=Qrly)U~RtUy2nD`hGF0vqZjK=<+E$IF;KkoY#~{>s;B#DGLW(CHZg4EyUR zP~YlF>6SQv7=xg@z#VlVa0C~I1qP}@;}Mn#Gj1O|%B6{NYeGh4?CL3J%4J|H0BWPm z8fphd2LdxQZnAM2jH9);51(r^x^NL(*j{?mf-@w2$cwX>6A&2BL(Fi7ZYI=Hqwh9NtBD?_i6uoK@U1nyc%}Re{ z>OzD-7aAjUm~_`LFxY{QgP(&kvAJx7X2!qQ` z=U!ySp>rgNc<;SuCb+Ie~uo~@m%++ubXm3y0jEKb*XZl7hQuCyh*c1jBAENUXY*H z_tU@{-f}^ttn(mj0fvQIxx;}%&Ow8NHtZaQG5f>%FoT{IuPXhiS~OZ6&h!?S+jbz_ zuiy_>t+rakvUA?ddR3;LE8yw?4)b!iK$*CU+mFPt^(NkKVHQC)(ay#}9}3iZpi+{n z>_+{vU)GV+u)o?;%h6cp@fSNpT)iRruQAoD6uFe-^ug$|h1aB1HE#(NidooR;omQL z75CPkX6VBy18KxJ(GT|zZAFT-T_$uV^*C9Wl)XG|hbWbd#*Pbr4)CDUmVGIZv|QdJ zXdT^HZtcj@7;o)XTMx169iMhQrS4K*uj?0fZB2iDqgCIEh{=JVTDs-uknxYqzK0S~ zmHozp(XN6cMD#N!Y$>hX-QQ&>DNpx*eeOc;{$+zv)0NyrdFwXCEA1IpNbs7Mk#iRd zqhe97>P&yVb7r_i2J#%-dlzAX&ix@&Wez*+WP03tV}rNg=mQ^IN}Z$4>X%`kyxPBu zn0=b4A}g70gh3b@7w1zNO+2Nki69rY^n*a$&w|_$pFaJ@xHlhYV4xxLhIn@Vcqe>) zb*&N7OF}|JB@Bn}h@Av$N1(2Q^G8h>89zA4s%qm7I@<|9>GQ3D`*aIk%dy;Xuhpwh znGLO(D-|xFS>2;r-24*6ts7)KCziW4|#{xLQWfG85lpP>5Kj+9gXG(O=60cV3 zcf6fa`&NN~hi8Ig)mu7)mSYbqOuv3~l!Tl64Hg!b<4GFv+7u>-iVAAU%IWh4d2z+f z%bxA@fE@AA9VJf0>JY; z^k-)$a#XZi4;+s)2S2Qt1TQQIb|!Fua;{gVO zk5SoeV$QgK&9;m3zn1ylOLiQx3H6~wllsbeBOg~+`sGV)5C}lv4u|ckBB%zwnC~QHCN#gA6 z4B2y+9uL3-&2C-$rd8`El_(Gk`XqpB{hM?laV!QHnNl8xwEIPoEw-yR3M0(M$rFRB zcEKaNH0=i?ubKTW{}tGAG&VA*ireaXDo}hvF*3enHr1H|VT({O&2DY=fG7mQ;nCsR z)+Z~YfwNJgh)Iu#ieh&?wSjr%3(>UKu3e*|p_v1%1^frB#1Qulj_PA3HeVJE^)b%< z+KdeDCTh`#sK3CYt>2Zye{CvJ5!o$mpUnum|^J)@Cc270j$n5G;b4=d_#F>Ij zqeED|K6inzqP(*397SMhW!0e1Zn@p7qwHSw+3ozo-_{uQ3c?Tb(9c=#+_~fNgiuL- zj>yT;SwT&$-PA?>Cob=jJ|2XDpw1Je`W?_mAbs?wJQKyVcwRz2UiG1v5ka zV#i7qn=28S?l8N_UsM_C$cQrU8rM_AuymAU^Z;CAaBZ_9Cuy*~g-<%B>HaA^P4 z);$!I?p$j2LOjoKy5?NHGODZSeeK+SY~k-_GqbJqVt(kW43tE(FfJe#jPpbE5}pZl zpxEn%C>fynhBdqn`SAVIqQ>t8(^LfiV{g5u@M_l%WJAl(vrr&HgD3}o3PGo$(+N-+ zhs4IV0Y(CWl|`o0C8lJ6#y|}MW2glneIRByVe8Sa^FRYJKqSyk#nCr1 zB7l6+zIO{y7`R-VESHQT1FKONNC=j`eea&>&1Efl`Dgn2RFK5;95mE-?lb|tRNLq} z=S%x{x>gvHjDT4L!~xCKtgwYbW9IMQR;Y9wSJI*>W)iB^ zl$7rDU{>|g*rG*55}}y6m&aMrBq!yalWgaAs}F~?xLLS0Q^gE2<;pX?Y1%mzYNkt) zn09b}^cp%H?k_0o2tgqN`fw_Vhg@7y5HvxE#V@ZgQ0BNtGc`5k;OGdKBo|#fjKGwOHyTF?*w!qFl)}eIJagxMEpDI~QST*%uv2 zE0dMPW@cvS*UtM(v@!<5tN6Vq>4&?w=ECry9)iq|dB|!Gpgg*=vx8$DqF~+&4rl!- zHSq)z$s-D7HRF4d*tRa1|QA!Ge(bhL#NOi;G8hd98R(POi~K9`f+p zSM05{DIW+5kxi~WQsAIWYj1DwTP-@qsUo0(w7QRrlsP}*`rM78Z03K?aWxHeO(s~* zUs+k&l!l$C#8}v2`|+Ca==D0_>rQk#-(5i=iRlQjfWZp_=j{_yN&?z|TKB8&fS2@G za0n-X4$$MECQA{wGq%#hdbx+=U2o49|3Hz9fl(`e$WpQ|>vs28{rZ8g^f6Xliau8y zoa%`utD)dk;a_ob)f-nbjOj0hwpiiRx0b0XH1FgSI>biixBL*is26NHXE)K&17`hM zaq}5@)2ic!9{k0rQ1+23ddmbJxA-xKZ6DyL>O33oVt?k!Db0CbJa%aIU*<>H^5UW^ zBe60XFf*=lLC9x;VsU)!$B!QvNG;a`CAHC^p-4!2DcMEnkb>>cDy9}2V-MPYwPC_Aa9=cTV(O6m=gvhjlX3oj);yf1d0d)k3tH< zJV7{B-_{nJyE#eOZt~(b+S6MUTEPrg>W;|C*B#CNJccdGyVCiM6iC5T-ds{JZ&nxa zFpSNXM7$Ee!;;XdKkTymil4fs<{XMw7uQXJC9(4Sd-s&E@9i>O#lU^KI8z65opKxA zgydx1aW=qUglo|RHr6^0;>iJW@Ru<+Cnr}_R8q3s={In3Mb0PpX+I_fWLjOrnGl;~ zmkF!D3Tm-v_Ma79IhNZhJ80@SwrG(J_$bg0mM_n(y4yuDp) z4~@+xj&Tub@@@IwpCexOw9XuE_+i+pSo<6q^>PTO81+ycVPxP5-!edK` zsv5u6=03}j z>gXP~L7FHe$%5EM2)JtK1fc~>QL-(Lx(m@(!Fl3>h#AX42SgpK7fq>sI7R07=ZOZ- zr8tTFua&EJ$CvM&oKBn^nkoD9+OGywyCUrqRj6)4)GZ7p;U5&@kpw|O%sn?*ii9P? ze)J7%b6(_=jOgB~3x>$@)SsvIe5hJlxFx&Ifd0+dbJo7ly8##7_zkKJs zHQ|^v?_>OHQS_a2Mn=MgTbq7&g0tbS`_h86)$&=--rFl4zW}9))u>+jk!1h3{GAK(xs@>A0wMg`aGmd0ZSyA%2|>jA(o`LKJS@3g^fzW_4+9H ztt~d6YT7kReRzVxO6c@Uewi^Kluk{VxK+HgyxecX#3Ia31IRFsMq>n}3r^2U|;oW^J5VX=Umb3EY6i2YL8sb1t2q3zEMy`kvRk&EM62 zJLd;OA8dc2KmHu~*^f$;tsU^mwSEDg+{c?XV?6=1dL*}5hra8%|9!E|bKo?&?=%U4 zPfS@Xk_#wX8*gwKjHccHcK<&I&C^Jgjx+OLkxU)1WO7{j z=k$5)h6!YrR-mn;sC2y6gsaTxZ60qr_`%CRb)Qb!qidrQ@PriYu!9B~YtJ=uQ4_@N zLG|>h%69+$9)$M$u2NOUX{2z>T5ctmx!XJ$`x8MWJF1|DZ__=&c4t(j6IRBj=OUw) z|MP<@PL|ZH;i#Ladslfx2oW=n8M|E^nS8EhBxSB-Vq@GiX!-ZY^RQqcEMiWET{sal z`y2`?ns%4>4tbID=?H`m2`(l*{BRKmgP4K%e}2GhsNg?GPa_a5GXIU=qw)A}PMj1R zpZ%XdF^K=h(mh7|=aC?^ul(}_5OwIX|9r2QLeQ+MdvA7s0C)p~G z{^z`0j~l571Z^wta(0)G@b0gw&95UF@|h9$ARX@VPt4>0@vA==V0uUtyPWK*SDQL_ zt6JM_qPkpp9ke%jk@4laqPbHgf@>p5(X@#U4~5Tnf&>l*!&NHnG%l8QJ9b-ID0^79 zs<}nwQZ@68kWWVRot*flQJeN0cd=jHz4PziW>FxI@L+?@_2f6La3sipW-Vg;Sx4ht zA&U7|&-(i6*?6t{Lj{H6edKw(d;Mp1%SEM=;5^qS!9o@M<0Z913DW#{n-j61cPU;Q zUJNvMl1y~j^__4VH>D25riokp_xbZN4mDUw@30V$G0a`Z$KiyoT@Xtwr<2ttI#q`r zRUMs+ukY2kpVZAkXIm-A?`9*;GG0K9a+s{N+7LdsuCX{b+B>ePR;k>a$X2noqj1!J z!!o`56(+}e*?bV8s9b-mwpCXr(^2WaUSZvD<5E=vaqWPgm(R~;v%)%@3gz12zSuWe zxwrisZz-{Ii%&OcbSO)KQ1ok{;l&}2dHc`E!j&UrY=TAtBvC;&5vk)u~r$FKG2!s92Rb@)fn3rnP8^*?1zO zR^<~>wpUq0uRt-|jkzgPIoW;ZjV3(hSfwn(y>TDP_KAsl0zRVueCD9`4TN;eVn?)* zj?RO|+5B}ny^+(sLHfM6wL$>1z7DHe9m*5ZP14MZ2E8Dw>&5B%G|DCn-Y$%vR0wpF zINWe$BI7b^pVmU+!C?vIDDGMz<|*O-eEwf%M*gP~46M6v7xO0GXwEF_l`AC~A-9o6 z=C!?BiCZ;csRdSx>-<{URkVYm#2OKl7c+-dCWbWj)2=dNDRQHsiF!?bmsj_GSU|;c zDR#7MrNLyvMhBMK6JnFrVbgJi9Ob<~dZ;8!d!dj$*0TG?!|@4`S%w_;+fOwe`GgX!>avo?yBu%3QZ!z^F^&p{^^=Okv-St9)I;g*LI0jjdLv(x zDz`E-F3M;nJz!^8+p&F6d0rRqvIK`p*PSoKAuVdbyL^+7OP6Uj{jvagjYRyM?!@ z>Syd8$MupS`?_LqvT#k=7 z5wqJevoqYBKu#<#owjbP@3ONa1UlYbw74ts3b^eb%?BqHv1ClM-(O~xT^OOQntqe~ z)blJ`^MDu_GebCos_EGBQmzCtVs_Kh&JhLzk0dQbADbLqyZ?>HVN^FDdKK!aGItaO zpc#Mu`~eh}IptW1;SLK+u7k7lCENn!UhkT%N@C0KFgTSUlp(4PA#zKT2@_MiNOhdp z?(F}Y7TK`IuzNAt^d>ic8Z(C7JwWE<{CJ)=$NBPdDN%A1&e;No$0`LF0`$&LrE$); z6z6MO5%;S&0dvmQDkb%*e+qW}8@RaNor9;CR5{_8Eaxka^_K9=O=%FKat_<~KV#zQ zGIc=WMxluwC#i2x&_h1HRo|ZQuU}95+v)y~e8YzE=iM7uuA66EQZkMM({MPG)8TqY z@!ANQwU2#IqAQQfF`et_KG+#=zg6*dEWP$m$jE=$1o+EE?F)p*cEcTJ)d*oX6pjZ3 zz+12GuQ08_Iu72Rx!&)Opr22GBX2+$Z1p|$zkSjF{EIo>ZMauiPfxUL^s_IbS^&G?2X&1|RMW`+FFNq@f@U3kDc;O_-@wM!9%w#*;DND{ zwWkiplNn~_oTMwm)wGwqsQL7i3;Qc6%zu!F|9KG~x2rC1Zd@mVW8@~F!1Jq zSnVlF>U)=JlvguRc3TP(bx3%1%dTJfr)2$K55)oZA>!$=yF%8`S&a>wY7P-2Bjc2N zt!~KybluRw13ckdUF{lmFg!ebj6|aKX1%fhF!FP!y`AO%ctanzOJtYr&^~b|)oW`n zcWXqjX!t?g=iKHJ2+!#JK8A-AWW7O&depi{xt`+P?F46nEMb0WDWHG^#Kbah9qul7 zYjvjC|C7n^zwdSla%CHCvKe0dbi6dr>)Hlf4T;5(wz}=C%%u-%VxT zuwL%!S(L|uD>0|IA!}86>)+SZz%M@d+}l2Z2jkykYYN@UjQFX8vqTu=5IhuOp%6Ve zS^}**5rR@{Yl7kH{VJORCh5G7+w43QWw#Z7+zsPtMi*(l7Sx@4eC>`{8jnd5w=_Tc-C_I|HblaQ3HJAZ(>4`6L z{LdfAS5p!?`-Th3|K--ulvE@As?LYCd(iZm#wq_99Wzi#ObqR=I3NR}-F<2Kr)&-R z-qOX3Kvl{Rn4{0y-&8rUF)07gKH+EM^#xgTb;vT&CvNZXLb55h6SV6LZ{y?YquEFT z%heg+3hx9-=N=yuBkF|nAD2GM^Ro`w?uI-f^Q5Qb8k1e_j98(!?Qf?%^|tqZl|$n` z8}=h7`#PO$MDJSkOUc*W8@e|d7{h6UAGwKmqg{SIj++Hm4Xbj+kK2AP@g4V0XXj?G z#fU3TRXUT&;o+mtD#3^MRMnI@Efk-*vYrvWGHf;#almJDk%q^Dw>AuV&JvXFS(?DK{pe9b_nh~&jn?7 zah0did2Vr@byegej6WD5n~Prl&YB(U;zssWyC=TuKN?U}?1<~E5y^G#IY@!^yHD;; zb+_t3EtCJZ=z`2RQ&dv_Zv?R8q zAc%dEdiZlKjV)HHGl8Gdxl}aN z^6573$?uWBJvzcTTp#tXPGG)pa|8C3Va$Cnib08ifRYjpTj(gWJLfw*w8N(0^ng0q zaO`{4uL0Y{2<4L*wk&r~2ig-8=A5Xqh7a!zV`ATOHOkbN#nEQLS^<&(J>W0Y% z`8FOy=SE1@TnwzcX_hvYv`I2gOE84U#s!YiiKxz&aIi64St4O*ut-Gh?Bdq7jS7zQ zaR*nu#D6h~*TcRz&M(Q1=`5Tn{ZQh*ph5MvQ$hdRc}hie#=wKxo75p8+c%kB{OKa2 zqNQHYMDtePVJYreu(YXfVRbKY3Q;!p*OGnE@GK9+lvbHGte+ia;Vi^CRT_N*>GF+W z|2U%(TbA6!mrs9Bjwe%4U2l+lAc0Te=xVvwiu_|`VR!H%Y{spO2lIvu&9fJuh#3S0 z13^VRw>JNLs%okAC7`RjO9usKa4;Q0+gML>s!Eg7c$hHq)zY6Gf^wDp=%p;y7KIFH z(UABhozaBy0E63nW|}4Q5&{=_dq!hCLGyx8~O^{wT3zB+FB0o=sYsx zz|UJ5(&-L6bJ?oJpYepVfwG(j6XwwrV^2(9lEvN(vG~*%170W zj?$}XbT&1ec-M9h(i5DVw2;WC3){3ljbchX`rWDBumU*PB@SvB#}P%Og{X+S%Jnb1n@g;VZjTQ;MaWa1YGSh@2nmM737Y>o5OZ@XM-PCRi1j znOpKLT#J+Qv9q!HW|u=6a91dXi23W>=4RBms1wcJ@_B3PZ7=J}yRebOYX}zDUI_0$ zzFs$}P(+Utf;*thqud#0Avmzm4}#cloHWF zBm~+^1+5F<f!AiZeb~S#n9$IDr z{yy8&FWrCsLcS4H)wq9I*l^R0|Z zu+n~BFZ2|*0X2*VOPwQ5;euy9BV0F?MS;iLu{kNcNlJz8Mgtukul4xqn#RX5<6GoZ zRB;YxyBloK4wXxA!~o4vSuu6OhDJtkjWJD3diwOy7VlhMH>wch>O;w8spH|Qe8_eC z%T%k5P3^{H_)!(p!2WkH3KSvHDQm^D;vfB&AyD|YdaRWdR&`T7sX0}gIgyE>0Q zEa)H=SnJO1Kn9bxyX&^>ulw>-@Cn~^r<1Vr=BA=>SZZo&()n4rQt}pa0=J3x`htt6 zC-XX&V{=Yj$G%)Xy9d*BbL z=p!p~Gc$bZElB_x2H=D*2G!Pz!;lQ#-68~Vmn`HD(lswCLVv;9y7M#pB$ir*Gvm2o z*?WNClhXwp3{n_=|BftT6&D}>Eal%~7XAr>aW4Zat*fFed!xvrE&Ba-H1Ax(4<cIDCNDJsMgOOU7ICyaXSvxBfZ3(;h8>+} zWO+6>MV(UDe4x0iPH{1_?^S+2$hSnkzIK`S_;n>K z-Ru-#XP%iH);~T;hZ$b7T4N_+Z6aJ8jyR5;BD%N5H3xF#zL^EePEz7ul$JC~ZO2~> z+7E$bOj_x+bpD6IBez{=kkWN5%+yV zKYI*0`TcmV#?=?m41q2 z@}+5&G_e*K?QyN9EDVJ~2oQwiYS(hWeEodwLC4HrGnLDC@kTj@`T_XCAH!|QzklEJ*cj=29cK?S zucS@5)6+{*W@bzP{9_dsPFmu#nps(SkaJ}XX6wfAuRQsYFlok{^p+*u!@;2M`xEi) z-=ohCCj9Ht1!KSSkSg88-=LOJf6I~tF4sJs6Vjcoa&mYuq4_SKvC#RGG)%LSL%;*$ zDI+zuH1@+zFs%H<(o!ZTYVC`0K8J@#*#*3p+dE@NWA*gUkw}45QU8a?uW|6srk9Z` zT_=_RHFyVJ52+#ow&TTA>+9?Az{anGE1a28&`pE-??;Wf18BJd>>tmXjg7)|{#!c4 zc6NhbdvhO%j*S_HzmBqM|1}5Vq#Hb{g@g4bstWP9qOdU|=-&0GAM#aL@xd}}VYFRS zuY8~TcyF!!i>d^S_HvY>zX%u`r;SCwvKU-;Umf*C#3@)UzjvzeU$}C#Jr7RTLGCL; z8aW=y2Js{DIh$v#{b}D zB<{!i&lzxuS6@OH2-PkSsKr$7T!T4RI4ovHoo-^10D#HA$ayLHNyuq?PiPXTRZp)> z*bW5AMy||EPpjzaf@>t_l#dlpFoRf+1#cfxk!`48^lcV-_OmvRk+MNMPiFm}iMXp3A`?MFj8m++psA*J`TV4I`>CrHV z?Id2bf1X;IQ3l`*`&-FZrKN6|J0_z=5k@WU$8;SGrHYg_wYP`UG%^D4C1k#vdZ#@n zot!>BC?SCj=j7m%^Sc*|yl!C3`8KN_on-m8CG+)FF6l`L@y-1nIT#8EMAwE#CYNdC z>syL=K1CJQpY`-*wn5Cq_yAXpA&?{;47T1jo)KKGj)*8B-_ZH`s@>vm`nc~AjN-#& z0r2<3@6daG^qg$GfP2U{-~Vp)*qRu$C=C|;q-4|zjo23{#oSnMgM||p&t6|$oH#QF zjt*B;zfRZSAc!0HQ4!G$ z%^ZenDy43Nz?~TOCty`z@6zzRKRmnutOmuF1CEc{c@;iuxc%Aa`(_!!qK}_rOTk6E zENLQ1M)p_OZ}ygA{JA{HWhw7IWdOKNy$Lr=PBt_&GQ)az+Zals9?UZ{&zzg@7F9~V zn+%HG2C>M^Mb$9%Z#JYcni{$#R^r8{KB{-26tyWr9*ZLQp=hc>K@kh&#;(&Rg3Mt= z(^Kc;O6MLK!o})Xa&mI#hIQ}ZNi9dfh!z3~)$fjLuoiwd{hGYC)yaLCD~@q`j*tCy z7NZzwWrUwC@Hs47Tv?xN0^Rn}lYJBmYf}G4MP+r7{et%X`h*mW$wOVH^G%|_hU7-V zMGR}`?Bx8(+J`vU)<~$XhDJp2DJGnDoYz*{41C905qjFt*%>=|jr*g~=+H<@--__@=+-uwahgSp>Zu@oGSdD<$cL(5T{@66%0A1TS6Pn6|T zO%v-n^S)UpF#)SW2t*vaAos?;xQ#Qlu%LM4Y6*paTcF}$DokL5o3tmfkGTTk?w?<7 zebvfWN}mu%EcVuMogMEtTkyK>E)rm$o*k|Yn|wzi{j)X2L5dwtR21HG zwl+J}-H*3~+3SZV(SMw$|5|#ijwx3bTT%+`lS4TVGNmW0Im|_6Wmcp6sk80os6&(_ zD-R%YVgP4!8wQV=7$x2T7NfUDj~wVGugX0o8pfvipPW+OxUr2f88%EwJ=#y^#=k^( zKVKVe5mHDQf{oCt)2k!T22Q@;$&|g~9Y{&c>A4ijOK z+K?xk!@|X5*ert<>ht+Jim&#Zx9i0U(EdnB)%C9{+L=46$;Aw>$a5y}$MD_K}j zSfkQ&{3>0$5(No@pftc^)@({f9-~`qAq=~%WejUnFP*9b|I)F^6z@ zLFyO15^C@QZ~ZvDlZV-hKt ziHhx6Ta{E1oW#w81J*l&SMc!g0zRCH14wlCZ`N(dIevZeHwe2z_13^imE=6<57N%{ z#DlW0kGPmqhaxokkLQ!W<&!XM{QUWIzfCCzSTwVzp%KvGuu@7^>c2?@xT(L#kt7^C zKGDLGsOFB2l>E6i`fO6lDU@XO!CT-C!wl?~gD14$)zcPwIRO%jw^Jl=?nFW-0On2H zY$oG@M*~2gM@42bx>Xhu)RDM)@rlCbmj?5+7@3%2AnoS^$oGxr-5XHiMk!k$NJ0`! zx8LW+x52lv6p;FuK%Cl#uM+C+Q+`0bcah3?8+;MU~*e4PnAAbslB;xxB zCY4ircBv$%rl%Xh9+zt}qKaOy!KN#%ZfAptW~9nN^y*|@LsH2?vrSof5b$pgW$|Sq zC_3V=D0_*r-n@C&yTh$D@SL41S;%VbSXpl<&jJE6Ynb;i20$+mHhktb-cSm@I>&jb zSGMvy7Z1jbfeOL<&w6MN=?72et)mUS4?bmtc(8Gtazp7ED2rwS}StSt2 zYoTV6t(1DSJW;`bUl+A~rT&Nl)4sEs2&^P~aW)Jblnk)&jXdB2ZHLfm4xbqZyn) zzw6A|LN8x^h;5c(erWZV;tHL}>vjvXop}f7)Gz{FmO+B?xi(jFk>!d9gug^Vg@wdB z^Ig~t9SX{M@!j~B>lM*hO+Y)lCn|b(VIgUEwJ&s|E?q{=6ah_^Sya6?aI%5YfOn^$ zr*hZ!av?nd4b8KKiqurvS|1OYERCYgVOAE_o7-oldaJ)HX@X!h{`=NO9DqCC0__M+g`g(du zyyi~5pD~=inhplfHP(9W6YzZacF@N4L8!53D89$Yhm_oWefQX9-%VnD>^TulInPNB zWRZg5>dd&BU(}?)ExJ__y($&oJ@fh3FSta$3@$EPFxUfz%{Kj2tex?NU0XY;LTDXe z4B;gf>ec$VW!eoYtV%lLL+%Eiea%3_c_dG2>t}+@(<$VG!~)#yn`2t)5zc2mJP`p# zKbH6wijn0&vA9_>Ll1~Kg=9f4-Q5)p$4P=r$8r68_e8ea7hMc%yr|@3C?%dUAnmoR zAvb7#%=yD!J}ZXc3PHEa@w>L1{?`!^Ipzy=|Iz?wcx`210^n=teLTd5)~&YBPW7Mf zpWiQ-#Gh_UpD%JkGXl&J>;&Qhxw2g66rN6Q5qF*&$~T4Wja2!w!9-Tm#6*-V(Mpl+ zj$lBdLrx9edp{0alE7XEcc$L535Jda(7T#h)kjdyUt`5aT$sKJVqypo z<||I+fk2{DXzZKRLUx|nCW`Lw|JAp$2Tnn;DJiD|{!XB~c1QyK>x4Nof-W1s3)%%U zQ=ghp#-`KAoH7k_9!i-9jiMGCbvOfsXe_vm7iM!y^oCm4SV*XSBJQM$POY!Mj>x<0 zy*1YPP_xZ!x#xq^y%(aPeh+hMqvF!ysQgUX$+0J z`*l-c@(G6SzMn?Rz?Dt}0-ej$6eGMzk8pbZkUJg-LxNI@?CA*P7k2o6Z$H{(49U+!1Dx z<;mP!h?AP2sJX0_r&b&y+}k`d0ySZ0H@7KBfidas`*xFO$6b1G5;g&6>Qz+K{HZrH z6bHIeBtk)o4f{P6Wb|#pctybW3XMpxRYT*{)YLe;yLu_FjVIg40!TM;Ftj+HJ$*t$cF!I!6)2BZI#rX zKJqs#6mnkNB%LTcq}$)$uQ4gY*4C!>u#OVEZp?+jpZU-Nd*cD}tJW=z&kS05kIj}n z$&Vb`dE+^a;%5Skp-h4ve+aAMr^#(?9Jqg?%`|M#>NLFii8C@4!uIp$;3yb$z=O&Otf?XDXZWNPWq~;&NuDjN~HZ<(mTi~E5CO+ok;tu=;ENeDA zR}t#5@o|W=Z{E7aphy$SA`g?*K`^Tf^*2bu5-rC;2q7~r3%4m+>H-GVE|#-JUPqQ$ z7P^o9odl2pqoVlRZM_UiK5IYB)xlhVx4p>} z+}iw|=O}7Un8QK5dL329{~7z~WN5p}-4U=|Fn4ixGnOubFUkf8EMJzUq0 ziGWe7l65ed>5_R2v;6OXt1AU=IZKm;*C8$>AgGjgcD~gTaTTjeDk_G>Qr@JX&*pnc z3G-cc0(o9MHel>?E|mJ85R=jFAmC6ry;$$MIYlaQ{6)ZhWhg^=F9Q{$ z`I9Am3bz&;YDdeP7W?cGku-xj(EI_GM%?;TGg@=cO`@tY@&R!;qd`FE{vq9PhOy+#f zv3v{CgQGe!gc^lr*(#M!twg7dUTnKX<+Zult(l_@qGa2F?-KCAfAT?gz<7--BcFM~ zrkyRn_M;djX=$e9@~7sop>~;kPt@bf{#7d9bOQ$gg-?Ku2%i9D>tpS^Z14!*#dqb+|EWFFtt{ zs=$a%^_K-f!u~cZ@!jbw0+1yE;p59=J1g|P>zFzs0Wq;BsWTJ)#&;;0-L-HI#3fn` zhSc^N=P6gG4>U(=PY&1(D;N|5(Xn{MUAD2bw6sNKg3+(tTwHEKRl=69EBAP`%%LSH zpca$I2n&ZUE>dO(IIX*FY`e(8SpOU7qrkngaC#K+hEm{K5ajCmWjxV;Yh0K^prs>H z>eK`W&?9B#Ot>x-oQdhM#bykzKqX31Ru)(C>@pCV^39eIg?JYi55{{eA*fb= zf`$Z>H(A=<%TQlNbWN^ZzRqoJaHo@(Rkk>I1BX?(R4^9Rji!v~pe73H=_$ zJfkD^Ig{l%3dSxf>eJ^vqv%S}6nl{KAlAD^9H?M?Fi?2$a-ezRkI7q_RTO;Ls;z^A zTJWi>e!g|-13X|t!VEVNs5hsX z!(7-nZ-DP%QHIR-vNGJMUs<;?AqC9U%)(JqQB?&C^MPU|teT4r+mbpY*F3Hxd__f9 zcYc%aJ;DusX*Tm0(ko8gir=nE(2ax9F0f00K~!iG`nqC9_^;o2h)Od`60`hZiFUQ@ zgm6O<5FxO2f74Zmk?!efPl8LA4%5@!VV5d*9M87wv;ttVK{g(VajmLc^he!3rrjl> z5SZM;kTB#`EHbYrk?-3Oj4pb{Kj_XqQ1r~$kN=8dH&dq=79go?b}4Ni4S|azyiguk$K0s4CsphT9V$GkL2ov`ipY#F@0~brDg_Grfq*! zr6mh`=XbF?4VM9brsBdC7?73^-JY|7O%%$G4e|M+#?7N7A?0Z-@?Y}#^<-xy)|Ex@ zfK~|(p%}I)>~4_uvW^Nu8l#vQwwHLg=cWD4FgP<42QsXg<498pTE4U=4{$Z={HU@~ zhwtDzeAV^_0y=+2CjI7M!pu*4iiR=oh*zqQK1zocLx&xNot02L+tA@ug(rmjEX!%q zIDU9w$<0r6$yb72=tKQ*KJ@#5`fWuO`93YJah_`w$}EWVXqxVI@e_7khF4Lve~PTu zCOOg1u9gHIrh&`!Jon*MJ#F>xrc&#Z(OvuJA!xvv4=m>h)Nmv*TTXT@y@5t@*T@be zXweX}GH{8fO4RzCqRv1o_c}u@Yx@LqIzMd<%G zAKEygttT0@!I`OCmBeg$LGI6RP7!gxkAw9EBgnA5NQo-8>)SE%Sofm z-_na)R!oThR39KP&FQ~FCOp>b*p8Ex!I{C#pl;*k`CI3?9|M})Y{#AfEU z!`WAX^O0c?U5BY5L%Q}ELY=WHcZ}h=DI}>Qn6QAK8uKZ`(T9Ua{A|8USoR7%bg&tL zwzz<&o#a;xHA`IwX4+!;$^)jo;klNf zY@iMWEg$A`D5Rz;W5CF$`wB~sRl**b{=@*J!`nB(BOa=Dp z+Ow;D9?&e8S5#))J#NWyj7 zRvK?55p}>V-rR;BWi;iXd=;(wb326~T%M9}eq$h7^6UjmClJ*Lk@}G;Ip*gtTAq&I z6hU8{*3+|hz~^}UH+QG%W^)@^!0p6gcXZHT3(HuOeJA+-3>+&va7>R(NFgrS5(QH=8>Z4T>FfByg@0&t!_fM)E!v==O za~w+w_NI&7$x26#vUmLcI$gKcA+D);mbP>JX>*j3rP)%o@vQ2RihR3At=}VciU5P+ zH>Xlqgg1JxLs-&dRGq7xU(7nH<0ma9-`=jb-@meVTIIzK8?X7|TdWi@?=XOJmnOBn zg9;cFBEa(%rzz~p)YKQlwQ)~A2Bf7F_G_^VDuPK$ zV->Dff{hO@DekX=mmMQxV~i~m)ZEP3RNhJQ1SuwT>D}GMtvx!i;?L+`Iyz2f_v5v@ z1v~qL7#1@+G1J{dmynl}z!82u7g&(-B$yaNR=OoBDo%s47myNUXtW=CEo4GV+vZmV z*3!}v&g%_}jP1(AN@eJIQ^vz|FgNlC--t9t@HvGyV_v zJXTpstk7L2S35OJZAhVZ2o(%yHm!eD$T|SjAKn6=X6436HLQE|bxhVXDp_Z*c<22i3UJ&!~WYT?2#I`bW7rL%}hzWR>JTdq{@` z5$|n&6Mylf#^B?-`~B}gVF8!Hplr=l)=DrF>;cHg$l+Xu%KG2GGXNvw1;S667>Zc3 z>j@Mb`~=q(!-(mdzjtjsaej15GpEPLVX}S`J+rv_y4>VE$MG5W>NOW)1i zp4=}0jm)9*=fS8T7t~>aJ4t?nOabtwmq=)6+MtqPtG4j~Xfsf%WCKAt`4LF))2mzX}g6uq@`nL*^{a;VmyS}}3k@hVmhJ*)9a8c=S@W%ltLER1v|3XHmzoH2oj zwj+VN?d*gUiYr)9RF{IHFjjb6o;EoR&E=r-a?g0-tUX{Y6Vp8`6avNJgF%kCT&QFU zJ{@a@2$d^cE=|n)i#GBSXe}{ZvY$U!)8mr!ZiD?$>(EdXfK|w1lLW6dw6(p2xO^IR zc97}?Lyw_BjM!&MgVlSZ_R>=jXeF0BT30_OzC_sE*Y}*y8ih5rx~c@Nx_dsW_n<=Y z`0K7jU*BC8R#tMK9k7u=>1k;BIKqaxub}6b2kqk>Xm8xWh)4y?1UZe4Us?vpKpm9=9ry=z$>{c!g4rbQRU$fVwW4tYd^7ESC z>n<}?zh8)AnEi86$w{3d6p6%@kKx)$J`V(AFvu!d*m7QNR4|C)VIJ;vWMO}*P0_pb ztAf^*sCIoVS^q&i>r;m{D8*xxRb`-{`ZB+}jX`UtUMB}la_k<7Q4f~{W`O`d&-0ww2PFYQ@y948O?xx@#ZMk->&3sbJPJR+2M{aBqgK*u zCVG0V)lO_0YC;ytQL3Mx_+gc`c<2v_+DDuD#k_A2^{uCpko~2b{l1W_nl}i_Gf-TN zL~XsC9Do0Q;OPno6o*>F-VdyhdIl3#2hD&k?I2=QOi={TX|WL>0e=5)lCd9GR= zC+HObP-|d3?DtJ#^e#A-m6ZuvSS)g)w)jC#PH&|=2hzyX!+tSPro7G&8{X3UYC&Mt zllWtJQVFvcpnYEszb{E<^R7L6Mu^?J#9Kk`6ZvMh^v1qlEy&WKU;EA?2?A)va>~W222;(suk`xAI~1_HXmKBDm4tHkqHd z&E#;zTXz<%wsT!(&-kZpNgX)@RnI3up#qW->=y z>&k?pBz#IY=DBq-FfiR>FUC)wgwzdkLBYgFkKE{56ZM211z#k!APYx_ml9A??8`hY@g)XC^6X3xEdg=;OVIj( zTBs{UAemJWeTYJrhY>CJ=UIDJ?Jag!cJ;Uff%ajmPtGCN5SeY+Qz7(DJI`|Lx2o+# z4Lh4kS_Vuy!P&Djye$dcXz)3Iu%gWcCzk4I(NkeTL47duQ?htW0Uai{<=!;tD#8I4 z1r;n_*F!bM`#}??Q@ztoFki>cx7mD3yT7$%HTP9ODgAClOT;y(^{eQ5-Fp_0)(u@g zUhj^9vC~H(LJyT@#Svn)ekCELOM6ryhP6TMyZdhEI2IN)zg2DrH*U5@iI9a1OTohc zfi>Pg`s*A?NO5XYlzIOWiD%fy{)t%vCii>+I z{)j};Au}_&@6LTOoTJo{tvT_xb3WosA0;Wtu8-BI+&|Cw{risrbv?b{y`&P88rrlR zKEMl!3(RE;i=W2Q6ke4n>QX%ygOGahG_albSAPYSU(&}HE$X@F4$^#{=P&D!sptmH zVD$g&`vB?dvT=cU0qj)b#avJ*5l8&j^h9U^WCbSYW$8hN=uk2VNYf zjwC+WD_-9zJxg;VA`S73%(yjw!WGyhg@{fG)`*!Qs6Lv%z(7#>0Mh>-ioF$2aE1y` zk_+pOt_7`+SM>ry4)VT6P;SdBDE&a85)N+HS&y_?V7jY#U}OcVj@X8e_t(ejn$B_X zFm;I5R$Nh01E|q=yFNIc3qe)-?CHmsz7NLRx1P!F2ERiV3YjN;KTH+}a&cmB2EMFL z7VV5jx13(Qe3v>he!&jPk`rJ~f%I|U<>j>*DHb*f+`B01d4{PA-t*q3$ZYciGoIPD z5kA1pKZ-{HyZ6L-szC~um=>!+RkFvd`THl(|GohhCEs~OA*TFbY;62ZzcQtv0r500;=j>EX7gnr44@OYl^}7YF zN8bVjf=4b?$o|i7GQ<>^(1Qe~_$e64s_uQ&YVTcvnD7GlrUE%mA)ZreNh9kgXtfT~ zYMYyzr~dv`0{o)Hg$s(qp2APpu&M>eVq(E}uw3&1H73IbcYyKy4*loJQG5~;|D~-@|}#P_q~u1dmAw7VCM7;M#UJU%%nM|%N=Ju`Yuez z^=9Re`^=C%zxW!uBi)fj)%7C! zBh&9u(Vz?L?crlrDq}`j$uCUPx(IPnAYOJUA=>2$LME?VOGU%@7p8a0T0wqI7WPcQ z?2AFw#%yV$lasax60G@H-ve;BlFQP-QY(=q0;%o~AOqX(xZL(RSZ<{6JhxbUD+N1e zIGiSCn;*{yPBwA$OYIUY2l_oVb1-AaHI-=D4tu0N895nO>It#v9|3A2IQgOOR$|jA zM%Ub0yf{2r;kLx<;js@qK1GG>kS^ghzna>ST5Im=gH@=sCrMwttO3Z(bNwRTPT#Wf za<@cCWo0nS zg-|xwvQCAYTlLy0@5|U66=uf1LRF_KkBv}JRz_Sq2WOhgw+yl=U=$(yL0fc^m@Sj7 zZ7wYAv?RVdYrq9#uxei(Uxqy*U+bLB3(FqQRnYd?QKj5rPYX{8V69Ko+|dGIUN+u_d3 z325^T^crJ+#BT`R^fms+8b(h@8Uq_EfAYq)YGA{$oP{?y$3&n(5)v5ADW4di zjIDNy;;{~8Gmg4+!mA4isR|sy?5xbkRb=I}MA|pMeXOf{o>XF`si`^hJLl`y*I&PK zoSq&RJ2#5W_ZYm&$_;0~+gDR^Gijp^od)kVrYeATWnvr(%A-HM1D$f%W>%J$MN;cG zGo!-|YF=%3DAWj0(^Ujs5Xjaad#?c9>59(Ih>#F1U?P_I4tr#c$j+NWwrABUZcQ;WWI>68EuQ_yI-ZsfaiNw%k+=TW!?btKa zeTZMwa(zfiw?G~lAg$3M;9mS$QY7dzq0b^vr`X?haKUQPcy%Z_-G=s-qC<&{_ieT zvxH=1CoYbByM2)^Cm}xNeRFeW?%LB!5)u+!>1dH;zH(E0hZZa3C$(E}FFcf>X<2M* zQ#bUBe?Qx7a9lq(zw}5!g;9g1k^IT0*_B;Gr|R`vqstIX5BVnf=^1vV)$VVfMvHCp zsEY5g{P{`nhKi;&=FpO*Ko5-V*4>od-QAUZZErHKS0^m>8&}MYnWT!h{qA+9qap8D z*|}cK`$QNA2m5@v6l-7NgU{I$rZ#&j;cdp#<717-ZmmVNFkdD6rXqSZRd00yNsA5$ z*j<|Gh8U!eUgV+OvWSYmM@$l5wt2~jZmr0>@88ufMbqEhm zyeQc&MHjPKU$={dyoe{s&S@#`&>GNYd{#Y7ruPHqd%PpxyvZ*uFE=mIze^c;csRE9 zd$_-!JT~`Q(9z{0c_3y&VkG$61qnML5=<6vTv?qm`yQ^4UW>)?1* zCeHgS|5a)6AMu&YeC%Lp?1CxmE_{M826ciER(Y&k1&z_u=j3{1j~{&_sl#B7g>wyr z4?G>S&fEPrDlZ^{e7+yXg)UFzgOK(8@Irr1vP7)yUDYZZhY#6w$s#h<9&1CR8#TA( zU8~1Ao{0!-BuVC>gBqzto`%e$0;Pb0p<(mI2y`24k?P5#PSsjh*Ds9=yH%{y3h0B< zWT4Ceo!8N!kZ)M@P`q%Bv5AM`+O_Ogukb+|qXcI$HSa7aL^q!hRX$7nAY^|t9hLK9 zyjlRfo|V)1)wS(kDfK_@t78fc4eQ_`G#V()m6n#~hsz|Kk>NHx&dd+=e@lzf;RJ&+ z4ifxZVq%7ccLMaPFJ0R0DhYgNODB09JQe!@uQIpiV?h}P(8$?6pR0D|>4G_DEqI{) zxYC^>AV>U$KF3kdDDQwOj4`4Gnh+$k_ng(jp8T{bPu*KM*kVgx{u_~O-!U7mUu%^Q z>fXVVOP4%u3pwRDR-1nTRK_>9?CycDgZA$~#AF;ym=x9toywA$^c^rFT`8CY$h$cg3R))XvWW9VXx1 z?&n8Ng&t9h*?c5yQhWUPJrfgiY2lYgF#~n}qV9E7vGd|PhI=mCJ3E$?U(aa(DXb=A zX}QPD_CH)B_|tja>jFZJc*1AJ;0rgg>q`ZrljR(4@RP8pWp4$QVv||#fXg_I^gU5ENGHfzw5R5xn0Artn z&I9+|5vzG?qPyFQfm)L^$G4c63<3G8UuTxZvJ@#aG4L&^e{U|qgoBGqc{1Rjc6-OO zeP-*+%p4WS6pQxPxqfMZS5I!V_?^PxyurL-n`y;B%~8RG_BoHd2PYmM_>a^>UxqG? zdrl0*_FHjqd*&GXy1%RP?2H+A)U;E0eN#1wL;l0vH3Fo=sBA#J$z>Bm26>2_dgBSFL%ru*5MqiW!~tANT}!Duy`o!8sdt>7VdrLXD!Qa zgIYeW$AsRs#D;Kjby5%7W~>l7X?-SFZ&7B+H$Yz zq{7&}L4K#qumI(J^-h!((HZ&Z2SG*5|1?jKYk9c>tVy}H4#v5bb@yaC9oxRA3o}~1b5GSOmXoazQX?CHr0T2 z*g?0tYxW^ zb9x?|aBzi~IJSi>131>);ZBrcW@cuyu9X%avVZ)L&O?2IExj*O4z>J@@bcwY+YcPv z+zH)Qnh(7y6goTLK^Yl0Tt%X9k>LNdD#`WUnScG}O)sos*Gfall$}ScY;3u+&f)vl zuXir&wA&%SVjkgouQcbo+T7e+%;(>Ul&~~y7KV?Cb(MruoQo z-9F*m+}c0{cP%IE`?R$A?`K)o*iW^yva$w|hHiZ_uOlP1;6yjPGfaFN!QVIC#JYXt zm($J1&;M9E4|(jTpRa9cWp%4onWnbH4ymA^@O9F@eAL|`di@gd<^Deh!&)evou2NqbfTD3$)|OEr8)cdqn>8IcIW{vX9Y=Th5Bq|f!#gwGrrm( z8>27R!w3*l{O+^}L_ltHbG&|y%_r#LUL`YwJK;pYY7FNX+!BM0^fTrD{R#-@PYj6O ziYwZ|Nb&amVxE>MFdfJ6Db!v z3B2fO$5UYi@=q}Ui|Nb}3|9fa}Bc@Wo z)+IDF6c%0w|C?uY+#hE;H+8hN+t)lwScLxb=u@Q&h|LJSd~H~+OT^^#>V?L`L!&NN z=Ii6v#&e2>4ZV*DLV85tScu79wV2n01fHsVlEsej;i z(ZSLIv5Fc4N$(%z#?sLdx6MLN;mTUiX+?$ZUiGI!I-f-;em&);F|;pc3>E z{$Hjc<{hP=;DZPXL-7!HC!UAU1uP!G9Xs40h0Z={Q&lsb=mALBa5D zFW_JX3yi9Uz6BUS>o33D{6D`{!_jM4puav!l`rm(I`zZ*8?00mnwy_L``UKkQ%lX~ zQ8SM+`3PwJvjs_=4##rzSAqY0sP`XO6L?@rnVmM0b6AwM02tC{I4t%hIS{OF7y2^WMma zyGVeW+Z)4zZ4p}XhNDJD;t7=D5dI+UTd;ra>e7$_9tT7Lzvuq_3<$*iU?~Iw2|Fe17Qdos zD?8dpC|6xsqAcGtlk$61xRWb$7P7(U{oHfXQE ze*GHLHbVqYu{|v|68btC9|WW%rG~2M&MA;#jD{K7m~#cr{5k z$-U9g&(r4Zrmqjnw*Bfb>PA%U8{$~Ic7^pOD@2Jha&jvlzH#btUK?5SFq+JX)-Qd8 zKSc4LJB0tfdx@Yhbr!4*IN64lw$WivAowsAOxCglD3W182{jwHQw|He;LYqFyS&@L z$6p?-j+U1G7W4Vr`f$gzvyHWJS{PBi94+sigvpHr7UV{+6lhobIxH2hs@*52^Lxle zMmD?U1xFAFiM~clp5FJa9EAm{?XEjjC2=&|J~+vy134{7}P?4iXLC7 z--rf;GlrO7sb4Q!xFp;D&y1$~VK1YlmB#>ULfK;_w>Nhj`=l)`vp(z}KJHFA583xC zF<{Tl<+ydL_ZdopE4qz2p)*TT#pfY?z^2sqoPR*Eh>uu|h#(VXH@Ek&ytE<6)-T$2 zhffzO`9|6+p+zJz>2+r;@_58I)MTyX+^ec0UPl}##(;cmcJCNI2RI!wM;+-iO11Nm zUVYNy2diSUBQ)13f483K7c~%RmElNgqe>Tf^peD0!7uHB8B~jV+F-gHz z*?6aL0#YLudG=ge1qGGdfbo?*jyrd><)ZHG?I$}eW@F*RjTB|_iW)OQrWHx!T52^Y z+Q>ok&#N?e|3$+W|AK3JQ49|QT=QFVBuhxhlm@3)#I?)oDn(QJDP~+PwQ7b6>}q4| z1zhVCo(U`@WHBV;EcP4pmkp$`OY5)Wb1m$Mopg2o92X@|PW4@{BttMrBnAa(i+OK{ zyd=iK0(TX`66G+iB`Fn^RN3mt(aeGP>UeEuvAjAA=)vd`w7GYeG@-L` zsmsc4I9G1Ob?(coNm6WE^to1QhFIY)EegdDLL!aN_+Nh<>cp}mn(;6MnVGk2@GOX) zRMBW-CkD@g6qG0uZJ;E}3?g7MR>kSLxaZ>49;UoWU)1{2UxZ$oUP%tWl3L9Z@QY%5 zj{82yEE4Z|bBO!yHTL%dd0mBszOdwR_N~M>vY@bnxep5T-^=#0SpWU%|N3PR?eLBP zKipLE^N~H`zF*CiU@2&7d&5X_$x*#P#u?0pQ{2*h#i5lqGnbTh^>%zw;N-^R?Lz7M zqgN1#-B!?;XcsfK_T#oI?qSvQnd z7J7Jap6c_jl`?@Ww`B0YPNgvEQNIUBCf0+8e``-f4<Y2r7=XPh8sW0Rv6cp4;|X|*xoe*yPmmV0b-y&()! zQt|oR%j(o-S&BpFNaJta)X&HLKeWANT$OA0E;_MM1Pi568cFE}K|nfGx}+QF4n;v) zx+El}q&p^vbc1wvclQ}{o&DSIdjIFc*iH|-Nr~; zG_yhI_~z*`b|RQ!>2`*JNVhZJv5oD@)zHh!wm9x5Qh4lBg=6I?>N3DuGYcK6o0lYC zWn#orlZ>(5U0ht0DonrdFk%aSmH7%uuY$)`peaTuDzejS*0gW%Is=`*bR(2L4j+b| zPr;G)^B3NEkRX8f2T8wl*(bN_M>Bpi@Q-?{C4>X}Dmh-DBbV#$!g}(=0N&yEuW@Ie zpdcB!nzzLeg*o`nX_zO%0$h8>t%rS*q=^Cdg-p@+=>M84XO4>zz9FnRzhLr-$-qUx`XQ z?sYHp=i|F}ttjQg^d}ydt#5j5iojBSw`vEDWy;BTkc23dJdk$?k?Ru$`xfI2te2V8 z@VsHuC?X?EXnX~$D~eHkI801TRHPuhIyfrIaJ&h22sSi2D)}?xSvP7UQhglOu47~Uh>!Cb*Fz`cwQ2tb-m4;LCBSd8Bivnvkj zp5g^iz#o&)PU5k6FoP6yJf*)B%ZMut5$7yU-HFM4ml39^k#-=TF7M`eYn5nQ$$;9-)v*)#wj0^{131GZEL^rT+;P92C>bp;>8jB*+w|<{ zME{Q#V|V2y>YTw6XbI6|v~4A87nrISVA9(WPVU(mKdc`{YF^0r9@cku2Kvl4$<~}j z;sj8%Mr4s)ihj8_^AJoLP>SfgD}&^~vi|(}RaI3la2?fEN|#ZBdL0qs$LCDM_J=k= z8JwP*`wX;oB)xhk*s+K9FD@^)`0Dw+PT=EZH=jrmZYvo$jApT<#ij{c94@2+9_N;L zLOeJH^yFLHL%|-}uLH?uRprcEG_v3hU=jSx^GT<)GzP&QAy9U-XTAD?3qZ~dR6;Lc zZJlNP)xD@vhK--5!OpYp^>P8WyqY)2T>8N2K|C_&meMB@a69h17HASboAtAXyPGSb%sH$cLa+5rem&>i@@c>zHYTdQEhsd>9u&_iR7+M`%h`ts4cg*b-N7EK?- zaf5ty&nVRBVfR0J1~_!n&X2Cv1rhaR+B$=x0JbYrHDnCvX1dEw>{;;Ws-5?rQm{pc z1jItU5d@kb3Aghv8|vH=aPBhzebF210+OS&FV0X@BOw)bv9DOt$vsM&_svX;x~?XK z%221m_f1gMs7ZIRTx&}f=r?BdO;TFT8=EzTC!%G;68zu69T2Sa6tWsj*?tVRy;Yd0 z-`U$seDCQbd2D$D#Ik6K+Fy1g&n+Lk%Q`Bw9wfqJH}-(AAicI@RqlldI_1~(C;K7S z5y0&Sdl67o&%p?-Wr7lN;ooK|eZ&QO4Vz(c7;S+PQ?IVe+Gr?IqagSY0sP^V#0282 zH{F|n83V7=-Tr+H0*pwf{aD#_^JLxA%JTAjaNF3JM9P4F+B%nwSjl)@-N*i{h$KJv z)38k6q^KJoVpa3hD@+cfeU!^a8qxfoj7AolkmxtCnML2X2!~QZB}+z5s(6!sxi)0R#tY8 z@cg86>iN4fNG^tg*!XR3Dk7`^HgO81D);lut^J;^~3>@5?a@8u>aQrvy;T9s6 z{^k>E+HZ=3%82uPF~Ogh9}lu2V^~aD&QGyESX%0z4on_Zs6MG7h^{VsXT3}WdP4Oxl8QC1VZ&IE~)8mm6 zBRJ02gwp$ZawI6F<1RwMJ+2PUfm8`m;FA0S)(EHj1xdA&n^dev#NJwe5Ngq;T8o;R znq??w5p^XcfjWRSE8>Oe(!COkNgt@VoJ%|ZG&H?QRiT4QA1z04%rk1=zmUwbc*kbnEo?c3*jm*Sa4 z<{n3zjyO2rF_xt36Ea|n%dCHGO3ctug;nC#Grn8Up8#-dG;~A*0=w{mj{C%Ds1m4< zD_w?YdTHAAJBQsjTN8z;sVRiOsT~>tztgvuS5K{HhMpa43iF@N>BC+=9ZZ_{K8&vy0dVHe5Bte7-QDQyK(R{K~y2PrO07dAKp5)6m*K zaMKE`?I~+i2K=SKp!>0P{6?Z{_21RC&TDcV{Glwc{Z*s-W z*I5HKkK2B;u8VBjF&6@IcC*2+hbVosIujjPP{Q!Dh-jW zsTmp>vC^6}KiNAOQAtqA(cjP1qt!TK)@r#Z*gT9Ia5Cun4$u&@7H)C5UZ0afj#}$> z@=bW_Q!ID1Ka>pA)IQUsO?m6bKh=&_`KY)WLk2FEdS)>Q&f-`JIRuo3`X1Hwez|?z zQzqxC-x=d69qHl`DxUfqnxw~%&DRB7PFM&)mVI}~?WNo!FaZ0V+!VxVHHV{BPN|BV zW_MxM|5L}Ow{odcoH@I)^4oqgNOGSCq@b@wh>34to*V=WqEu%kQZ(@O>O)d#l!^>n z@DGvphx9-M-6T^@dX0}o%%j!Pi#@=y4SnF$)YG=E)Fjy=rTfqt!KqeE#H_#e3gi}X zTt%*C)xJshx+}<4OUxI+CjxBVaEn#U#`*R}UlI(Jddra_7ysmY67W_ffXm)DQc=uW zAo#%bz~ywIf+eE8cpyccR=w3a)euZ7s$5_bZ9Be$9g%T^fRL{1&28+xS=c~$TySm) z*w|oz?MB@slB9%*_^~6*}cMHe{r*Ayxskd*IKfSz_qw})42$*13w`Y znuGUU6a2T(9j_?y;lp%5+r-3EB_|20<)P8h-M%x;>^iH}0W&f#t{81$Ka~}SvnW8v zw6*Enz3tXVgf=J69Uo)sxhBAD!9?~IgegN+s=L%ZJoUC@p0NXH5ON zGtQ#hhFD5Of8J)xenby-drG@U9sXPaRj~kxPp_$S{1N@zZ^5j=oHvWZURT)k5EgeEwkQB0E)wF)Q%3-9aOf@ zx(v#VCV$-pSFPzk=n}OT!;5dQkB^SNLD>y%c&Rqa^3cfsRtbU~Zngz?o|)Z| ziX|id;eI**W;>O&9u_B%%oia2M8@-E{}xsdkccKj*0T_rE*jszQObnqIaS&m--a$R zU;et?+K`_5CMbCZfnf>cVVMLzDa%0Vz(AYjF(1E3`dvmws)mEecR4dMn;N`I)%t{m z`dw1_>Z$VE${S;^ghgjzXFRc84f*!1ITuOqs#85P%WAcEq^k+CTOXhXsxrTuJEmpV zTIO&E!6SGGfN?>$$?E>z-XCa?)GH5wb52d+j~@^j`2_N4!U>_^jqgnrf1$s2jYZ{d zwUbK+v@1WCG{-As_E+9T^Xaf*5#LCQa=z${t8`&zbd+uLHwqPdEjjnQLBeBxk$=rG zuj5X-OhOvi5TsbkG}YG%oVYR%RY;}E%VjCt1@mzS*z6Fwd&^>yum7or9aBX?RRojg zh0fRnV-XMN+CtV=A($O@4@|J@dIi=$dg76g?z_D<%)AORrylT=c>@NM;Ln1})le>V zOSU})-6f(qgq#^l|F=17GIgJ><8XI|IM(;~W8>q;K3YDmC;h#>^#Sxs!!N~ZH&-JF&Wu52Sp!)m*DM0 zz*Giaug{8N-NSp_cvvq2x?N7N?~7(Smd`Q9MidxyiKI!^?5&I^Jfx(<`jwOv|6;v~ zIHZ4=EAguR#>DkR``lcx`~VT-hs_CitQfBm6I;x@iH;Y&wLJ+Do<@xUgg!}BAP3*6 zpUcRwt1#$@`4U|3ZMAw}rjT8rd!55=hb)fUPOEtetv!m+Kmm)D_=f>Hn#kj)?%@4h}BPB!w)a$r?JSbEgj{4|~cig}}}w5VRw`Sp06y zZ2+hygaO)<54R9%SP{=)+`jaw)!6%B-o#|ElM3zK*>O5~RGTi-?s%zZb1Wr0yw&^8 z>}B1~kZ1_AnAL1yDm@yuUR>%e2g#UMk4~eisD@MOVTxSx4M~>^c3wDN;c$3}_c(%$ zlKzkN@$RAyw7F1D%ZJ3j&}m99!#a82tF$xU;RohS97l2m2#*r9956`uAv1|aoz{&2 z8wAjceiwV#v~6oI;pA30-2PiwLOxAa@eu(bVRl>)0~n)&WVCs1HP*Bm>3$tLi(Pk( zsFm_?YSkDFjb+@O_1HpF6jI_gW{3cp)SLAopq`aV8y#t1+uNR{47J7db7z1mElJaj z?IXJB_ zAU5Ysu;=4?9ORp5$urrR+GT^sssaiXIg+UIz1 zPUhs8`Es4?Ew?;Lcw*38TAv-=2FCN-`yI{g!`7V62Rz`bn<{k5hG>sI?ZY2U={JUR zgXit@FGhU^03lo2z|1YTl#lqDge;`A*E3gBJ`exBs-58eoe4IO0_XfOZRp|-% zQcFX9s{vDm2~N1`o;R_mWD^nQ=F7LRu+jyO832Qt zS3K{jifC_pY^jH2O*EKbq5<0a$CS-_=dTekkxS`;3#u~Gj(oN;@EYy1@a0fx z0m%?W#hWbS!#pgG6M@(yS0*kK}-AJ)1~QOw?Mf{s~tXw%~qaYK@{> z2y26`*On+dj>XP|MQ8_Izz-JXEn<@qVmeY44%G>h?$+;tfrxz*#WaosVUei3ygXgP zr-CWrAVk}7^eKQ|+zaEv3phYK(wxxbaD& zRooJsX2GWgy61U_jYfPHeRY;|pqQob#eAI0`e2RoadlWQ93Ru`>v5)nA3O-n9)2aCwfhc>hz~&LckOY9h5KcRQ%_M4L8>n{m|^eZ2R#uyb4E=q z7=U)yR8?L$T(Zv&A&D>HVuCTGWx-zscv}9XKcyB6QVBhaXdCAxti>n(%Eir1etI&$ z`UU{4*E!Azx6bPM}=Sm_M}H2HZ;k;fj<^ydi0Yj2;~w8$>W*In#`jH#dBFjnbm10p-Rb-@AGh)`)5nR zw(*^c%%9XN-3a-SXNJXWOZz*@FX8D=`AjB3+40wiIfu{<;Fd?=O3elK5#ND^cCbFO z(^PYHt~Jijh?$m@K#JlapoH!5%#&P0LOF6eOfe25hY>TR(=#*vm6hCBL_9^hwTyVt ztjgE?cD}$Ch1qt?G!6 zI2GwMFAr={!Q$uPKK}l?Q&XGSXah)k4RH`;XS~gDDL^1#F9TDZ=5Nm+gP;}fLo|bh z1|l$l%~SHlhQDvbA6-lA)Q~sk-`v7Xu1Q=8sWjV9%}nk)7_0>b7sbLeMuqXEhxgu z?JwOj_xm&6+{sb!^seOAv|q(gigzxnOi%vsOb09)KZwk+TxeHD4PuxY=By#^BHbi1 z@H^K&C0OXExYbht+68u!r-5B6x3*sZZby`*aAw@WaU~)+8nlFD6e?V9;93~PXT}6| zsm*2&Wy96v`=nk4=IgfsHd4%0Z!%=*Ew#r2GGLk2@pFiIf=M!y{@$&4zPpP6hXOd# zQ=n%Je%CK0C20qLO+sJm)p~JZ7=Kfk6M&Nn8!@oAO4-%C^{f{Ki8ID_z}B4yhBsj4 z-4GDV-V(t|V_ZBRJiRgQt_7onGB{yxW6YfF$0F3K!T6Ay8xV4nYT~o8*n!;u%1N`tm*-I&EH9bI?i6YP@oc5`1#N+yZqIK9o$@CH{F*+M&4;Cua4E&XLHE(tqb zq@ILoP?l=?V;E(Cb>HRs!D4%~ClFJ17o(Z?F7+837txp*Ao4Eq_m|ts%w6zdD6nrJFFlK%t$1FpmD&A8;4iFigoD*E&uGDg|7M0XP()88gPtT>2`^jni9aKMm++q zYRMvQG7#*7!{R_?&=oIH;=21-V$n%pHDbU-E_3clio{}L!<|UluJn2@ncgr$u`Se<2m3 zQY@;{-etbkuRrN;;UiYaSPN3ff-O%{#5Zzz(%)V0qfx7wC8VE-~X~ z#M#Rr>Pt1jDlyeSDdNj4XDO5mbQ{}fs%@PCI}r^cN;l}*_?xX92kj$>hEQ_$!o?Im z(Kvud)Vb{cOd=n^f=F50#hQWWUZ+D!y3a?3rjs>(9s_jBMO&1=)XKdk64Pa0^25D9 z-gyUCRs;iIJoxvQufD$8Ge<85bcE`EFZC2MRN8I)(BC&U9WIK0lP(iVr^>do*w%J( z7ikHH6++>ciV$?$EzzXh;K7*q%7@`~!bB(-@`Z%yib<3kZa)Aaf89wjO*vfM2H(dJ z7s22agcGL_DGCRf5ma#t&plgh9|Z+P45aM^M}5;G%y(H>!v#eEN=QhfkX>>k z_Sa-yFj)06n~ms#cp98I#LdHXw{dGa4~TO9z@xp5V*L^{|+pVT%S4Qa*nk6V2?Vdm0#r zdWL|e@M1r;?hUvWHo<@a<~9=hV_@11EdX68#pSM1*p;hSkM*bSfv~bQ1zl%()cq*} zI4HD7jzl*+(ZZ8Z4geK4=Gn@X7sN>KBMyI@3tmk(WiOz1H~Qnno2vRLn4x3bp6Q+@ zwOQ*9?BZ(!zo+B^ckJbDEQ0JyGIZ~mS?rxX>9xH!4P#>RG}-`)&pM$Rb2lr#fyu*9#$ zSZQbazF|jmxZ0UvzRlIgxxxkm?~D*WvW10(+nFT^ybOi80rs!Fio<342vTiyLD9_u zJOdc_&y9eVs0jSv+34bixg--zn56_h>0ILj@o3JBx-BeI@M5vyRWCtkmVGuzm;ig{O zqtgT?N9o`!&Et9!2CkOX`=`LXIsAr!m??$@CY!}NaBJvNSV;-zV_ER=<)NXx9{Nt75Nrx)1H;4X6`yhMy(epWjfVGF400;ifRF0zv zz$bI=&rqjP9$7~wJLGG_2;tG`qO}e34or_F2Yx}gM@?Zx!|6l3lQ5W8ye0K&NGE05 z!9|g2$jClJi2(kKO)wt9;p^>fIiP(W13LpbH^os8DGb;Js_5f^WwkTc^6-ZHDQ{h! zd#b_O_#ys)+0f@gzdM}P^K*?Iu`bFrOn}9L+0`dc&(QNr^&$uK*+!W^J$3Qo*lo#u zr(p;BA>w8V7O zJO?H%5d&!JC3*$77iXo{etj zn#4=w6&|O(uP1k7tuBuV=0;04$zYldl?Bz(qPOA=1~#)1L4jDzC=KSkS zT#Gs_X%urq$t5UI2rO`K%|`5|YCj6W;Z&OeHAq#g#uj*m{MRUEw+6DxTnjpo^l`|cV~`jhC#S9@Lqzc`K*b!?LNWs2q6_^ zyNyljA^oIlwkasTxMyZbE2z4f5Y;wY55}J{I<8FCEX>Tpf|mO5fe|QfEgQS*YZs85 zSzBwt>9E8IyJr%dd1uFq%Ep6cQ7g4+-~t9giV-VC55c*J{78ZRr1!m&XW8qsQ=XZqejU_5lz$u8i1` zy?7RUV}BLq1fsbU=MZNNWY}Wg^sd#s{2q!dq5U(*Q-J1`${f%&miyim8|%F_W?kp9 zn_tSG%_wOD&Izpd`$Rr}h=bx8H7A?viVWJBubBfMEt}JdCtPuQc(0*K zh7UAgdQBoSH^t)eC7|jxM^zkP?51;Zc74w|KIf{|#@I2fMpYQ}KSzq00w1x z;)~*41$pZ9lD8VW&>8Ktwf)stdE3gsph<@`S2dQvPHc^)*-RSzzNxxCQ5*pL9Ql-` zCl=q6Iq2GBIOAeiv)&+IKKxb16tr~tTq45sGw)qj%H$GnH*TC<42`|zQz>6FXFD^4 z+Qd4ws-u%Iu1b6J+39+)6{!{iC>@8_j)!}$Zpk@SA!}(emjRj#@SebV^ko-(#(-=1 z-EmhlR-T&`A_F1r<`bldRTm!YFOP>7=t)@5I#x6^8+1jnxu)sjI)IzJ+6x^2T*3gl ze{oSAD&Grv%T_zFX1>kZ@7bq;0_eegqLQP!bMBjHeY=s{ciR-(az_t#cKUa+7x_ZVvLJd${| z`}Muh%a3x&O=u`95n$Z}lRDecYgV@c_JS&3ZL=aidwZiSYkO0zbzr@G=d#6viXGYy zGztBWfbPou8jA$qxB`c5`755Ez@Z5o=Hn!znJkZcu_Nd;KlT_xP2jz?we^TP4G>P{ za;v*=($t%>Ni-q<#Z{e2ChemCa*t%aidd3GF=w;%rIe6vq#${0935q9f~IA8=xme=vL3=rF9Dd6Uug4%Kb;Ijv7N=) zL;izQ;lAsM{l>?ZtyqxQ6c1)hJu_MQ5FUepGEGT)G}{zF0rS16A*F?mXtBCaZZHOu zEtRH2fLZzPQlT4`TT}0=m>qW8J|(=~Ty9yCtEW~;xdo=st%{ngsp_K$7gy#2^N!oV z3y4c1j4sqoQ!b<+;5)kMY-h2!+vVrJB=KJ~VG9=95lIw=6(=nM*o!wMF5GHza`O8G z1d<=mjx%I`@SGm3%dArT07#5e8wbvwVe19gz2MJa%JqO;ygS)|xHF!+D}cUqsK}B| zE808Ta62TI_a^vM4VFHEAwX~W>L78!@C^9|%srqiQzMR$Cm}I zD6`3d8iI$HhibcF5f<5FNP`nT-qB<-JE4WNq~8N(qwqBuw<{drcVKw~+=0hp3`)a! zuQIhStr-`RDTjM=b@kpwo88go>fA3}S02TbVmzgyTDq!^{_ijBQ0NdB=nyRf6xiM{ z7;aj;E!7|egzXK53>Xa1NsT5^-JZlE`~=x?`0VB+jEsy@qr3K~mTpr^#!!h$Yk6;Q z`6A|UNeT_aY_up1IA1W>lyPx6bGFEO9p{;Rwc^OZqy*h_0un_DA%PD8Eq`r=1+)Qx zCKbAw>F9XcdtvZkxZHxHZ?22k0AON+?r)nAT>u=I1WvoTr>NurjZ5a96y-Z7`^2x| zpaEg{%Ul@}(ZIFM*)TZR|4lUigA4;^Gdt+m=1(_8!8@5{QU^iqwOU* zM&BftW_`cNSE8ai7MlWrYr{ZAAQ8R;vkDP#e@9@qM%FiqK-Og%RB0DH&I3-SFh&AO zL=#e87#OqhL_~l?#i7_Ox#nh%8I*n9!|}2ww_;h>k$`2Xn6=K)0)OQ99tU|Mm3%Vkcs{?%|Mi7vDYEFrUPwKbtyTnkZSUZvgvwny}g!B|o6tgHtEpX*v7?pxo~`bnUb9juLXPFWxg!0UcM z2N;f0mu{}m`Dd%-U-L*I50WZkfkUC)w&m4ry$qUXpZfhM0_ql#&OemetHQJvH*a}rWWwSND|^z z;#UVPy3@>16v>HY=jw0d5^TTYy=r!?(IBx>w<|sXGUq6>G_(5M_UbTx3(7rmadid5 z#DZV3+y)e)fdo3ECF^CW)^ND}O;3ddgX&oxJ(3jst@%a(5@^A<2U?+m3pGO)Jbw|L zOn&c@l9DhwRWDFAfbArpO%JMLtj zOYvES**|CW2@_JD4Jj|ZF8o|?Lf);f;*u7m4;Lk1yKb#@=|&-i7v!Gm`}QFBmF^s~ zbR1vDx$|Mqvd!h@A>8y7nS}UH*Uy_1_yJ#l1m0*iGYXiH0pZo*roqlM^kd+^9=_tV zy{W|(8)|!THA;#CT9uv`FJ2*^5h9087Y^3BsRvf~orrEwVBtCRbveKz3%ll}H{JhS z%L%v95%(^eQ`;1OzA1g?xa%S^SyN4|SwjHX94}Ly>|4m-ohd@2^YYTcUoJF`+rCY0 zaVDANX!OU@{$*$4qyM@hK!2Ph1z`m-bF?Ht5^?nENa@wITfu4CIOw65>(3Q ztw8&q{|Jlg=rR<6_#bZ!|2xAk5I1o5HED*FQf`X8v{^>g!s_LGs(9)ds!UidNiBJc zfNr4h!lYBN>UV!X_*B?O_Nv3YgCY?dCLrjVBSH4 zm7#&ZDQyt3HzH9!gX(SGQ{_V4y1~IGnwpxx9frFF=7Dl4 zAKtH?VBl6c;1L1$5tbkD(u8Ll?>c_CM9a5q$yJ0&^IS*hpi?%u)jThbfgXI74|SPO z_ToQ#iz*E|IyiJ^Ee<+3I$A=0o9?>%kJrKjW5MXE+Mt>qK&}vSiS8mX z(v3;?Z!0Y5uyCJ=9;`uLkxN;q+Z|O^Zb=GQ7t|HLx4^!kTRr+$eSzI&fGL`7OJ+ z4{f;!*SYP;p()N$Wee%-2r%eMYL4phwzz?@07a7W=+5}Jmw=azpA?4{GIN6>q;k{y zsw_mjyQvp>eEUZEH(MPGq5;6FDvHsK_T|f$Fo{c6F!`xo$_GtZ)!I>yCot$@YHI}= z18{|b3*TK}%PlA*RQ_Ru1x^V-B0%GC>yESd_(=eim}jDc1xnKdgo~d z_9roMUx6bowR*`F=(jG{@b~V6ehD>0h@gu>4I8-ZR(wU0A#fAh!C;Hx`3f%Z8g<7$ zi+$&}kB9b7%fjHM(iIgL!c!}ialov}0DRz~7`O-YA)u7s>n`EF;Thad6FCh{9c)Wb zxS<+t5I~ri3Bh1csoEG9YHE@O@s$45*LD9(cCdxK9`xVm!R6!szish?`{mW#J4CN@ zk~c$d(|4!XUj2N;kg1tV4|~0Cdl@>yW_9FCAmM0VrHs3qdcpT}nR?Wa7qF3>rl<}I zOkz$E7-FEr90vr$j^IU~E_3NqrzMcwqh+Eaq~_vMaJVv%`ml6!Y&vTp2W_a_mD%n3 z`7Kb{dEH&@%e@qUfFXfm<9AUk?+E}L{0;dOfM3M0m;vxFS&--rz(}4^pMppbVdR=C z9=kb88c{>L`48`R?9bl1xkR<$X+y1A_&4&I@r%oz5xVEuT~Z{Q}O&Z%HkT@;_jq(_^=kxAP@|&jxXTd z{~m%~|K!)hQ#}P{X_rUNC?Hi3E{n^=0vP`e_*&lQg658Z#u(E#% z$Sq{CP?9h`sOIxA88nBMfLguck;XD_x#KDuWKv$C9Yz*8X4`K9uq?rHrU9t$P)BaJ6R4jYww{Ks81O~?N zvjFx!dcJfRgk=98$Hq*@1=jzv3Q`N}Cu-X6YG-6z%Oasl(Tgri)YgxdFfxzQ^Q~Nv zFJ&?-R1+_3U^#RZwv>3o)M28avXT=UJ+Gqdb4UA%IaMHxkBiG5? z!K(Or9)r$Z?k?sP&&9*M-$eLB~WP+T1|gfb?`q-U@r2i81BR-e0yCh6Wv`zbrE11zrgbi!4D}j=h7Q=$U0@Fp6=i5b)yGv)onkk@Z+eJ!wDmngcj%0o_ zizwe&$+Q{lm(M@HP1J6<_n1Da{tx=Bk8dWrfUN92g{*#7*drnW8?>MC&}RR&-s)|h zCL<@bBXn7;8{^&wlj;Okq6j#$>~7`pm;X!jmGX!q5VC_9{3 z_)}t1zw(X-MM6=?p?WZwon$rtfy-{Kb|=;EMYN=33`ab@I#HV7m)o?OB~c;)8&iQ^ zFUWqm#c~xJ_pttAZi}PEs3 zd-CAPATC(aejjFD+SwBNtca=8uyf70-#1AlVDxBa*9>fnk*_BChIgpPo%bka&)g$4 zAvK7y%ecX~_{z6n%lO1Z0Vp)pY|VZBQ`AS<+M~INJ>-%#^#;B>BCg%MKKp@#s+pm- zYcR5^>ar-@@mVOmAoDG?d6r(Aa|Vt|@Rx1B+|?0?%KhIz%?y1Q2om;Ms4TV)cC=T* zd^qByd;4U1k`*-vV^$GOA)%ZR-~GV|G>Dh$+cOz{QLvi_WP7URrO=W7AraqrML@&$ zE~*A(*6(;P_xpjBRj$UH^T~zwn3pFReQTqo30^Pufo^9nHE&TkoHx35?fUf@s^>Ec z#{FM4!vvg8sP((P2YNSne*UcI;OK1G*1^8Jw#(4H{c+~wTA4ZTaM3bpyvx}OkwAxF zEdG)9t@lEf*4Fh+gNvCQFljLAtH74QbizyT6N3%NC%~-9Gj8Tkd^r zV=Xdp4_^8u$ec;Oe2HeyUha1npRKoAm0)!sCmno|5Pup=7Kie6FkIhrNT-JmGpAc0 zjt5wB5Io`{s6tL}LIZRQhRL*#@4U;Jcr_DKOzZAUzFAf1btU$$N5o+%DGiNZthujl zLjgG;7QN3(2vzxx!(Qap1VRMG$=?=%ZAo-2ScD!Q-Dkat-ATyFzr^!afMs)RhSHBK zCJ!d{K&nWqdcbykd_1&nH`2t{99?K2v9~@ch7oysw(AaF%zpj^+a$4UuI;fpn)TZG zQ>%-snSII5>@~yHhnadb5GNSLZvWV1QdOqa=i$(rodzLuOr!{=4oDp(9ZU!w`(5Yf z*E~Brb1-YC!NA{sz{xpyf$8_Zm(JI(&HMi>9*_EItv}6je{%-EapR3knQPFmPO4hJ z<1HKx`26_`Pz+REc(O|%4XZOx6OiT)P_DCQfMMV9&oCIEq zMHQb{b1ccyp!-vB>_|zA`>kI%y*}@{tfb$r&rYw_tUFnA{?{jP+7pWp+N)~GD30hf zsTi0kb%uI+byt|t@}F&yT1Z8`lJ9+w$7U#?qJmd_x;NV?V$W#wn}b5~`=@bp%E=h_ zcRVtD+w7P`XCGZp53g?ZetrLkLA-YQ!#x}vj}~1LmLih!@^UaU(IDdWO28!4zWC&T zNysH>V-uG(a?L@zTMO(FDI;hDn>yC{k0sb*13E3oGgNb**(T|r5_o-(yiV$azq+Tj z&QI~bnhgg2{0WXlBu8fnnq>!*k8s73MeOrjSM@LQ&F(-@JPA|MZKy7$`eaIXHe%kb zjh|!lRx}02#QvzdL*&gR?g`%RwLv3k5EKU;(hWPA?|X>!^1O_ccBhm zN!&niv&_h`aoXQ#BD|{;#pHO!??FV_2;Pge>ounr+?a&oBxGbbP<5t?|A_KGF?jvz zKR=zfz<|9QgX5XGr4xp5pJ~$O5e9I$fss`3_-DRq5V5!u3%&6^Bcn`@J46aLYM4%K?YE+MOh9 z4Gp)Llti+SsWa#uffa=70q%sIacyv~^T?jX1Er{9H+nWxRa*Q}$!I zwZ0W_%s_V{As~Q|<@vcbzbfx|ckMV}0bEi}Q>7y2j5h zB1fuVso${IHUg9Pm(e*eJWL~ zsaKbZ*=@HL!sSThM?*t{8_UIa#CR8fTuNWzF|V_YL9K@6jud-<^#Ns%G1RjA~3;X6Lt^`q8(b)wSBW_E*Zs$4s4xx)*ujkSReAz<&Cb} z1gv6sL`E*U_YaPmSBIw^jtP|#RU3`}V;2o$bL*q6`v@ZUH95PrqmWG`|I&#+kjM)S zq-S-FV@!luf*FN(#EQK8L%G@J}Z=`)j_|DNzTkeNPoMorK8mlk{d&!mc~`onBiScbR#s~3 zdmH1SqBjpHfmxgpN{QH-?^LJt*ZfmqyUFa>>i~5|r+NVXLnOyX$d#|T&}SF&=4<YU*fXoR^dIG@mXd&6nVtP1WWkbl#WZE(_LUXNk@m`@_U*+} zfM)G>dS6Rsc-!$@Uk)7K2_LVXxD9StVqKelZd2W z?0=C9z3#)zw;BG0%`J+W>lF*-9qsKi%LefV!y&?uN&1rS#DUq|+`Q^ygl2AD867L@ zm6q01UT1!*@j(VsQqT2K{$xlN@WyRx3 z-XV30l{IPl9WmJ&#osTk66tK#oFj(@XQ`=eoPLrXj*t5PEwTZF$yO7KDlxbc{aKTO zO@Ub5xtFyM5*&@YXUA+6bq+T9YkNOCK%{ukVr}W9jP6t7D$P;l`{S z@0^TxXx%@qur&_Nb9jbjzic-?`0vf=)i1mK*+Tcbsr#d{;^jp-(|G*&5&$yMt?NBc zbvZt0-6=i@FJZmXsG2z${~_R_6`O=K%j#e(T0VUUNmNr)Q)D=S;fcm~dUmIPJ_ot6 zxhN`aPEQevZo$By=t}uvdq&vT*LQ1sd-`D11Qi?s;Km0I0wvnVJiK(e`ugEtd)nIW zVA&r&Xet?ApIcbaSW&zdN3_zsK$l>(Bpb3?2#m3@9!9UJDJ`hCI^Gf2PEG~|V)4|1 z4gYJ{o+_*FQE7-gO?wGxz{#M2;03bK)u$X(U&qGOZkw`!VRbVDRjr*-&M8fKY=;sL zZ@B}|`xjfiJF)DwyCz)E_WFLvr~ks*I$YGYV}oVF_?;sN(2ErG2MZ;t zS^tCuC-4Q-PuBvsP$A<18rlT`r;~qG6%P=^E-o&{G-lZ_FSf3NT)=O5c+74z2O?VX z4K|4g0!bPJT;m?{U$%cQ--VOm==fM}wnPQusS^1Ex^V`^h z&CMJ^_0wgU32@1Pk*7%FGi*vhk%qRvGC*#*t}^I<6S$~KJhZy%b~%Wu>Z%XO8Sco~ zNbe5)$~{6OR6vw+Tv-Ho(SIOE#UkcnMMo=N?y^?MK^;U*Ya^?4wAx1TudV4UIWO9C#rcJTtyJA?JVwi-poOrY*^<}T~{D7vdmJ&L`DtwX) zO<>O57HEVJL;$cDA=FC#{=0j`M>tJE>0!2p^YIA@{KCSW@((Hgb85b74fQWfk0hfc zR=6U-nF@R2tBa?BGU7kq>2u&d&T|M5!dn_-BjPyqewM;?<3P2j)Y>QxHC2O8*IFJ8O2p0#payH9;} zL*M@Pog6(vum60$ug#aj=m?U??VTO^)qAf}+W-4)F2;(kdHn0Q;XhwubP& z23rZhd;Is$yvP6LFDHBD|MS776b8^(6?aKf0beU@~?j9W&AQ z8L}%MOsY$~e?PrRknvw%>-F)EYlveha5wb=nGN)yh^5hVb}XB+CJ_H?P>*Hp8+ z9?z(#=ITuL4!x8PHAseshe31aTU5lt#k=-7&2J$^K;YH|^m@baQJ7kNOFg%L3cbWVT%%ov?ty?RyR@Ah7imX)>6 zprU4IT%45lj0(KrEvzgx5xL7_x+s3XcQlb;XrVUnY9xXL`jNd zD&_`)0)rblrkoVmAJYZHC&T^~zBHKx!F3fmfIrzRhUG=%{Jx$Yz@}9?so@DEcI&w% zxscGHNZZ%2juZha_?wB1ip+rQ15yTt%}g$cIft2+_uMQs9C#5UD1-y3&tYK?!y@vC z@7)s^=PKrOUBD0%^JX;(KidG{{+A;lnLdF+oYWn90S* zH-UObC}jePWy+0_Erux?=Y~AIyu+3D?iq*jk=EAM$uMIbx^oA%0&n?EALyx29jJIK zIssRNVhL`#+y<$YHCUjm!sDiwkyJe4IOKZ7w41mN=Hq~X@iC=^0^u;kRP-* zvTvIqaSOaN!NU5+jdHC>6f~)gkXl$hDU|Hwf z+*}?bA6gbGG0YB1J}YF`PO`hb4y6bH+L`Y%^Fl~$ti;>5QWcH1DA8cuO=2jvLKgD? zZ~5xC`X{>2K)J*Pq8j90xxGy7mW`kbpoGXMU@P}6lTj}-R-a;aGhHPi*{R&!-PhYm zDgCL*A5*^jxIlN=3IWwvJ5(M3K_H++mi*2VV7oeb)AurYZP?&OUyjKW=&wI;#R2ys zCf$q2*(i~igd`J^$5@_wd%p(<8(P}AxPBj~shVuhrt7_cszC|&3aY>acRsOFBG>%k z+ir`@&)kK9LI7$t@NIcG=sIjaNvu1SEF=Fm3 z+oHT-(<+`5))KQ(w+^y&8;DPb}k31x{(DswMTtBh~LosT_0-BZ3(M;etK# z-)Cx$Dkv+(7S2iU-Ky-qn7AS_ef&K%bT~McyD+*vuKbWs(F<{p;jOOjqhjO1WP!S* zxi-xx=nK`@L-$bSCH(uBs-yn)2MnKtgA2^|7DDzR~G2c3^yFgBO0w4 zd3CU+jP_GA7ju22cp-G&4@B2kz|p9yACFt^quB2Gqzv6M#2ZNf*_o_nr~=LNC01?s zkE=>mhV<{AE2g~5(s!wLWfn<(L1(XcYx^4TZxwzii33-t3`nvjgKpw5UIEg&UftQS zQ5-LfUM-@5Nl<9Ai0S*74rrP$<<_uDz|J6+Pg4ds7PhPGX=-0GVZ;yzRz0$1AD$ub z{^m-Y+L;aQG$@8;$%YS@am|1o)125W$O2)$zcoFFesppqoBQ(3^S7Ce?2~RXVAo`urBDUgXUtWG=gC!Bv&qf{l;A)>EWGsIPCA4g#8}r z&m8p_#j(PtWKb?KRX(;HEQN%{+Yo~%qQh|0hy@Q83Mt|)K6^5J#_tfZQRC`_5_2G0 z!|6IstDSfHx{&g}1mD5@=L@(YI-I<<;i7{@ovl<8R(8YzpHI3N&$irV`Su&W?_d2S zS{H4ine|8<8s8Fq_Cmcdv;Sv3KujReySA}$(Aoi5<(rh(?;q2v2p;?PJ0QCRfK1%t zJOBSs_TKSazyI6+OWKi?vPoqmr4W%(BqbCgD>4!hS=p;05t4|k?44wkt?V7LXSR^N z*LA!;-{1ASectcu`se!VJsMu)e4gjyd_0che)!NU57+oV>n@y(&>8yUeaHV{X7>7P z&2F`n{h^Cy%tf$E{LHjqC*HV11m{S88lH#wTCj1vC!GRtfQBeH#rRv_si(sa7{)+D zg=5OIL6=P-%i-`&3filf0R;-`a^5o?{bD!74uF6HKoXe2NGd2C>~+p1&)3}Q%D550 z<%;3U&Sy-{ThwA)DUlq|^uH)6xsuqLES0@iY0o~22OYnzZ{Hb9`Vr&sU#WNA*Yg`k z;&ax_NMCX&xlXoIGBO+MgsDJA9Gvgf4SoX&b+`X{KiADkymmqKPt za0h$x*D_2xeI;)MKwSKoU_M;oTb6yuFhZ@FlZ>IQ)ljp1|0M7}`2$nOUhF$(S#jxz zBcBg6h=M6e9DDIJ>8zA{2Ce?Sf++9_m6p%ijyv~HNL9Wq4LU6B1w#bst;l8j*NnVx zv(uj?$mb^xu-)E0vuN1-h4#Vbs^=nO?pp&M!C~st3N$b(A}8Kv`Y06~c{QpQxG?I~ zykT+)6UjB(T$cW<`#*Yb(EA==Q3!!tKhtN%CT9;Y%F zoHih6HKg{g;?Rl5wasOQe z6p_3qp7#Z%6B(sBj=?0PFCPz9$iw80?-H>rp(hoe!LvwYrr7Q~f{!R3yK$mznpz52 zNYv@=Xn>PFb0d4uxyVSx6-beyEb$-O*Z>>7Xzy!$yrXHl6Pu5?T`BA-pNaJMdC?0X zq)v{zQ*2a430B84ANtzA-S#W@mKDi=-){D88Ebkl+L%I(j;}S3=+??7n|!OOj#u;o z63auaTGyqeb$T~380w1bFRr*Yb=dyBCCR4cRC_S=p0H)ni0Y6fCMtX~4jPH=E}U9} zjm@i~;u_{j!x_mA?O&j3_0kZuAWp;sPT##2<+dhipvqU@#-k~S=SF@Q$R^+u| zR8!PizKfdr_&}NSHJnKe=hBi@vX6Wg*iWdKue;clyLGfwa3R@!=Z<}Y#Qq|p_H^Z> z`(Xpn$g>IJQssrY-?gxi8Mh6c%Elp$90yt~T^CmgoIs(MRMdnY0yghbM3!0 z_Lj9Rdt3@F)+6{$@E~1q>OTS93(3(N!!pF^J=W>~i`kNs=!r?{uK3XKO-FJ zJ&v4|*?8$2L@)5LqD(fJ_MCaL;$~CKzM|67YnC#`9*$0aLI!(T^mY?wo_B77+S(fWqyG3jlV9;9j6xYvlGXfyVKx_5 zw4TiLv%)pyS*stVwZ&rGzD~>PL0xR%pz%=^9>ZS8Em8s$`{F=vObqxltjuu`E*7kW zeYz`j3rC)K@ehplSlQUzb}*ktwi(akpJ$&qpQ7jh_Iy3-@9XTPB-hq3oj{{ z^YA{4o9PV8C!-ftGZ}tvvk$#q*I*VF@1Gey;HJKvJ)9&c(zcguZ;5_7z>=3C$kd?70QZ}9S=}RR)%9t z0xuT(BORlVRjD2iuTvKXHii0UyCMuS=^(Ez!Pcm{St-q_6o2Vch~NdsRWhWXM^o&L zv`#fKUhZOg^|(Q54mwwkJE@vr;I4yO8k{!3HdE{)Mg+Ab-bs2Y7e{n1A=f;5;r9X4 zh(}tJ_r@9zf6ErMzK~0A~`-= z_c}=lj)MC14&b*0F8@jG``ygW3}&msm^lRT#1MUSo5+^>)6PgiK>_7qF*b6>)&!Uu zZe=S-?mf&|gy!k>_h(%2p_zl8Q7(k8cUv<5>cXTf#02o7IAm!b!iirma)ekS;!Q0N zRD{2#75HuXlATL6@>>*1{PNVpu8hNpi9(ybfH&c!GwC6_tfYjS45V7W>q70lAHl%= zs#GUJcKpah`?# zJ4xA}MdGC&i=&;8mBGzi{uFfEfL!`c88P4sYU-^B1*M>WEAGn$f0&-B(IX485jxB0Pb^Oe1f9bZ|&3oI!s`=UP? zhXz`7&J;5l^AOGI1SPjA_Yd)6Y~ay~<%{u&Rd9YWR>SJJdmqI}ccf%nmgBCbXAkB! zJ~0D1LqWLe6H170GhUp78SLonU|lKTf`p0ykyGfNGKe`f`s!o5|4m6bSRYOyDX)?< zaY!yqxb(ZkB46&4DamUUDOBWw>l+wAzJ_Il*P-oz#@)Ls;}e3!2qak$q6Ne zvCV&Ige=bzloDGW-|msRa%J03NJ8Am={>IMQe0^GBFUTB8ZQCN+t8`oPHj0iBQ9^` zyrvg$gC#%RsFlL-3paNMg#U~?l9!jSV0*h7)iA~~+HLPst~fiF{`E)U>XqNCjm6>x z=-|Ne0d8dI#U;iU%zl*s2IU<$_C0Xy5uzsIf}hs}@tkv-Fi2LW$8ikH-dp;~%RboOEHpbJU-mlmnz{KwqAjnh zOw~O@%??d60?ea>S1v(+dmt zm;Q~ERuOsI3t zVHm_EYCSuhXz1OwpNUPvZ9YCx8t@ZCWL85?0FTnfs@2N!z~N-I^fQul&>@mB^1Nw^ ztA9KdHrF-=Pc_-tGeX;RyIOg^YG}v#1eF2?coa$QH4YkLPw>i2x`RIFIXZHU9LpCJ zqBMJ4a9sdWHAVdac&+;Bkb=j{M+t=rSwoo_ms2>_v@6YSsQd&wJMT)dA;h&YD&?)M z@23j8_wV)Q>Haq3$Dr%xxfRP^7)4+vV|i1LZE3dAGv>4i{{?=}LhH7kx*L|%CH@UgzcZy(;yX#Ve0g;3Q6N7HW&)*3q$Dc2o zzGcC~Jq#YMW-10w`9yNm=g*IS{P=)ZJYgkTRTLSGQ?#G`;d|<5(EGwMluNuzY1}8WY0dri_{9<~{OEvy*U!#s96q5(FjISkr=~`o3u=DIdSZ~DKbTxeNzm#K zS+@Cb-&}pmTLDuhoCV*18Gd)}ESA94!yNgK%^vqv9`ToX2fLT*Kxx`zpp@F$l;y}? z^)5-~g>*BqGAkCP0n;Z~cWrw5THU6VH|&wxa?&~ogdeQ*yI4B%E8(su%DXhsG7EZF z^!b(#Z=F7^&KGrg!w&Y}cj{jiQmGF1%US}XgDsCgxT-do1rtM}07N5U8v+3ob&gq$ z{~{V{T%N)$4r_&$^9DKgyaWYSf@t_VvA#cs29}Tf1jJeNXX$1|) zGd0ywW?*e;3NCRgDM2OoR$>6=sQ_-tq^5H^gDO%{Qj+pOuf`aC)qHnG{neMDq|c^X zBz8-GaYwgcXAlq&Xh~UlKBxzz2P~Vk@B*BrnVy?FZZpgv9q{QANZzXHPejo8iB-B` zKmQeWVDvbpusl}AQwxbKXHfQHO?!JF)I{%YW@b;a>e|Vesc4mg;2UGUNX0RgBYyom_*fk`nOErdX)x24oqxOG9iSXrXVNY|+p(2Z0Afm~T_Uz+$Mb^CLMay`nwT0oad)Z*AIOLcd-m~XbO=xc4jMa4| zBn@waFEtYI-d~?O58N>4seXTYv(r7-i~fAHV{Xet*8=HFiq7NC9CYM~iIHz-bK+Kl z=j*~mh@^Pi>7(!7vU)V7+!%SerjxqMW>xJW^Qy#3_%YIPOSKxdLyoKaqi+X!yyH>| zs~0edRVbZ(ck=EZ=D6d}Z8olNC(NWG5wmEM)d9 z6~A@wmWS1$g9mY;l52Q4Rmf%iSA&Sv12`1k^gGDws0fz|Z$Gy5UdviBO-B&tjJVB} z6J|tzP!~Z`{(?>40U`;2g@pvk&8v%(rDKW@44PtDKfK(rkELY1xVJYPbSfybi?b## z1`3_Nkdl5vQ<3-S1GIhRYCklq<}4qVivRXq!fjw^XwRiMxk<+HQ$_x#?l6*n4{L#r znpnSi(KWYCkI~_DR?l+!-et>NnJK>8_i1i0J@e^6&%#16$5rw{{fg6sA3r8Ubn3Le zRg`+ZC$YD6oEzrmzuE&2Y$_Y)^3lWEHwsy_a3^u@0w`(>DoS6wg!Gv&To`xY(!Hccv+(cYVC>SLUr5 zu=nx8pY!k(fVIAiXeh6)S%VrM6eS5O8Q*s}*)3DRkD_0s( z_*A_+Q(4nzh@;pYUPZxf=BtvD=lZlyZ2N4j*Rj&wO15=xH?9>M0xwnbcEY(`myzJt zzPTXa@I@zcz}Xoe3<1M@#s-J)J-@m5&zxz_*tTuk^!)V4d*8RMug=Ngv=c!ilx8j@jdK%>dtQWU7hO)kA6j8 zhEr@!5P_kX(nX}Iyrq`@a{?zUkQNR(p4FMFG)xgU$9ywlk}j1tu#13KK}Nc zQkuB!Rj+lIe9(aUqZ7P!>5d$6(@X_qN2ujIofjO6q6~G#Zm`o|+H^ovP1zU?mxdm* zkb*oFaY3B9D>S>{ZF-%DJtC145)bduT?7ah!0bZ-g6sJ^4$arPaOz=I*m5bj<@7Bo`x|*s|3i%R=Gc1UafG)5y-0B@ z+kD4W@6XgZhb4dGHf7woNscj5YL=r!`Tj=+jo(PT=$p|32*RI4Nv<{YX%9=-@v~^-=4|W=ln4z%M#tH{Lrw4GunqnMvfMU}ZEt+(r#(E!20<1Of(G_WC3|MZuS! ziS?xx)(Hi4=h2GmM4pt?jVda#BtVy^Sm2=AGu7+|L_C<#cZ)V%1libjI*W;w^OyFV zu6nwHig(01Rwj6+=F8^;+&5Cx&eL4f->zQRRhGB1FGho{L;~9LUH8_PuZI*qZZK?* z>Iv+fT6aYcy6N*}M@uzEduDGb2H>aB(km)r@#Yynz7;Jl<(?7SU=YY>#!%XOs_Zjm z6y_`Ox~c{SG&p<=s7B0h<-8pFC{?_fos+|3{p4&AZ@$D*m?M1q4vFFS`GW=5%PG+Z zag|FNlK*-XS@-2C{aCbEO6CKrKv{n#`X@?%BJaL}CsGo;rH=g}7u4Ix@X ziH*UB{+yr(?jRt+2#s1$Y)Sg^bzsQ{#9-oiRC=*kn&pfGnzfhLyt}_`6|LAEqNKgP zIMI$1-;C^rMwFTo;0cWNV(`bSWb4pk?ejfs{Bbh5FfNYWVx*=oojkZbcgeck0aat| zL&xET0Y_A~s@ug=qVe zK_mj{s&2^zXUe*x% zh+3+;3P7!h)SrIi>n&chxv&p6aKRo~Hk>KgSu=DG3w{;4m&o0^S#zJ(pGet+SI zw#V0UOuDhV@ICxRZ~-|&Nc;?^$B%&8@HEpYZitbI*&H+zDJ6#v${H^_@VbcbEfs&Z z5!;wPEjH0LQ**#{ZE142(=-cvR$7`FYH&d&P7tU4Vr7G_R30p)<=5dW_YQxo7MsXv z)RIE%2p9lyOn`GR!nsfsG?kZc-l%uw*?MAmoi+U{Bencf`E;PqA636anf(auKa6Fh@ceb)(e{-__*9Xyov2?tRF()Imf-b zmbR|0z8vs(UfMW=TLGk);*x-|2;fh^^=8^*-hG7d!s{XeYQ!CY^URg zNxuVIzv3y2R_p1b1dmdU8se0vBBB1&OKa5xI=tu^#|6btcjz(b=!0TSZ4RDULL|T^{CVO04?Qp zKyR$ZTVf3;?M*kaRn_bblS5bHJ@;J}F0R+&JNEHBncv;jHQ}|Z=66w($3CP%?pU9I8^S#V%j5&lV#OeG^>BX_q@@OBQ?RLriB}I&tEAj z2wQK>)$3w{mi}Zx8V4ls0LbJ5UcW8>PKYIoi;x zdA-Gk?yrWJxy$aI?Hm4675z#VOGbFNtlL8xZ<1dOFgch0aQUk!AR=O~(l9x>C?dxocSr$3Do`eeT_-Wmat ztH!s-zQ7__^~o#YesxlN)6_o9UWQqCH>*}p9`=?$fdbAw#u;jlJbJ|eTMxd1Y4u%Pa+{5Q ziKhzO%T()4+`C&z>6N$l@JnW=>uFrGuO3U@G=+8OyFl+>hrhGT&g;$Rrc@>N?q<;x zp}0vQHAUp6pdZRujWr2MFe?ww!nAgO?R@uA?r2LUFYt!vgJnp=(Kuk*g8uK4cYW@M5Cd27m^oSDToOnE_&Rjv=i=B!~eb%=rQOhCQ3%rsvrw#NQtIOVe zCnW|9VR2wFG{yBE1>R}*o;})N>eK{^zBvAJ=_04H^X7d4X1SpKmzGTK;yuLZz3OKW zU0!C*|I>hKxNbk_szlc;kf(zx`aD6HN7HohC zmupP7b{1W;Jm2exbGgiMsm9I0~kQ$ zZEX_xA8FKE_d2{t?6|lg`$4#w^4th&w#I^ifr#^~ie5N+^LD;3JP!Mh3)qlW~xEjw1-Tl ze8H#{Gtf&`8B_wJ3(L!PxTVSx+uH&ixa$wr_=Zlqb0D0XaUh$P%#P1vXOrSMkvRNx zTY~`=Ea^8D-dcECvXpd6$x}fesGJ1j8NANlKeoo%<$qCO66jE(dI58_bO47O;8;?d z!XI>GuIFp|goksZ2*trw4Y&OTCTYMEGCJP5ju7P}fQnsvc-i8%5}ktQSD>XzbZmw9 zr$gP6+Uge)GBK|nDx%JB%zujIt4Z7i0US9(DFM;!StJZ1Tj^lP8_RHE4O?pvX)Sb; z-?@fv&3u5J6L>A3D#?Vq%*o2Z1NO-rd!fK4j=kW<-OnuA|LrUBV`_Nw=YfcHacSue zC8f~1lh)P(7(u%B-0S-pA%-4`SqJWtvvn6~E8hz8QZPL?pIlv_y3cJbAW-%0v=<<> z&P*aeep!d9#x{;G06W_sHr=p1h$%UqqLJATd-;`E1qD^B9&VXkeMTbbXV+Zh=bMcW z4rP#p>Cg17qKMn}1(Fn4PO`n{zQ&_98*CsH|dCrX>Qq8#p^jSBtQ z7N)VJ+M6~4SY%Hso+Bbd^Sk&CFy^4R2({Lj3$qSmV->Mw{3RcG#jJ7H4ztBBTSUpN z*Km~9=M-KpHFU-u>7>$^3m7Fa2wMim$B$4{0vU-uGZp7;_rOtuPa63FRgUky`=|0> zh8J>!+i{$WYr?c%9x9&iA8*cx%`MDCAY}1``dh%oC}??RPamda${8+OIAu@hEPlTr zICb-;m{GeQ@gExs;UAR4KM>&?P3yJubHn**(Il7t%tezF?K`VCeF+5jP@S%yQh&R$ zZ%hg;T$6DSp(Dww*ojkMCw?AwxF|Qi1O2j#k#vOrvbN7;(TP2wB7Q!=m!>N1y2SFF z*N`x+eQ7O&%TYR60y$yZ;r!%-nYB690W1%+0LDt(fU8gAz*#fR~{hznko%S#Hna@3_lhVk3Yu*l~SnX>cA8xQ4 zF@VUA4)Ktpl8i78YW((%CUlscm7QpgLZ^GnOl`;BLERtA4KUN~0|-NW_WjAGi-VP+ z)ic&QI>&ZVY|Mx^9{JbnHQk99RdVB2oZRWg9m~9CjTuzcdBP&+Xy)s?TJD`aYjaDC zcq+Zv_nwB|p=Mvwl_T}B9D7-%j}&A2>R>xZc)a;TQsxyK#&#QG=Lx&X$#tiu*0$mv zZDK2JCR*r`dPz76k}%-T9~1xVTO<45pU40AN@jZ5$O+GFD)FWe9gsICcXe}l;3k!% z8P`5CGB7>55)OXV%vXZ}@vL=^ICOQvM^v3owHxzeAiuMy@AAfFS0Lm_`-X?5$CYa< zLs)s=pV1u7owC+WQ!!}!c;+{yFBNAXGX|G1XZaRx3cZXp_^g`ZT|ustq#S!Yq5q0h z>GaT@PE3>~QaL2X(rZDPDPa$aqe8 z#9_xOdreF1VP3H9Kk=wDORvub!B{61JrK?i*=&fc!0o$kvRMBO*}i>QzR-!Wh>2XM zZ5JuHiwCpKNAwnx4T+zP-eYIr2`Vu_5)bA-ZP<{vXYAS2WxW+06y^cPn4uK7R;$*$ z>`+XH3R`^B!B%U`?B5GMRqPdUF+?{RVzb(=TP{0{RKJjm_5eFh=2z+J=omKV90D;; zt}{^P;NLB){>e#MFv)27Y%Z{{d_@St-tWAkCH3_bF1^lTMdi$?SCq8N<}0b}zwXAY zU{_m|*gNh!Zs(0eiLLSF#m>-oTq{i1uUG0-yb1f(rb1LV+4~e=2!Z81){$bJ*z4+I zNh17h%f&db^JmX*&Y_7mSpcojJ2~4>xOEF`I&s|Y zssmE+;{c|aeY*sr**~84_4n6E2`rkC$Xh_ldqsFJ(B6Of*ag?>J_dASL(@la3D`=6G8R0wWMUg)iMTk^{32KyZ8vkvEoyj-4M-4#-)cp>FLol?)Vng*C#m9p3CiJ!5PNN!|B0Y z*i}~(q)5b;LRgS(A$ANIY%T~{`+R>c4(bqGDOMI*MR1?)PD7f5dXR__9VT@}27KmY zjjXUXoK%QDn{U5#REUjEOiK*aDO7=%xm1OHxC^@K@CQO?XS%I(121Tk`O5p+*x5Wr zUvAUx!Y`M9FJOYHQR~33MfL0Vg)*&&Xq}A|f}CbWGT-z2kFocLhdH%;`PFmYh3&y7 zhotbd!v2%!&PYTx6uikm69M;^g9j%>G8`Qpi5LzEl_2uX&O|r6) z2Or8Hh*8n@{3Ug6xbzfsHQO%fL^^&Wo}9V4^t3$N&DIu0%D417e!XH4v3p9oi(+7M zvNF?1;F`3wcv5r!-@n5KCy2Y1H$!{2^^~ZuuRIzrm@c=LzqE8E=_+u14{ha3`E7aE zu3bASDA<1TCeljA$0c6%PnrFhrxlkdR@mrq-zp4jqu~k|EYyEPpT3Rf& z+a3$}Lc)XGN_DPVaoNnwiX<*O69q@~^hhYw@UVKsPY~<$uo0Ez|lxDct{9qq|wIsDYZZ^I!jy_@6UVm{;L1B8~@S#<~9S(H!6oJ$a5L zCH<&oBvfMmk+xggP$_j5{q8lsG3E7~?&!N)+afsPKhB@IHkB_umA}+O`KE{Ju;hU& zLf0HGZyyt3*)x`Jy0)pJv3vW1{SMpak%CY8Pk4rEg4(+$>Ow|@rMBTDlLc#!oW2Dy zd1ie@EKuVksSE7$29Oz&YUzwQ66@*nUe8Oe5R8p2q27Ft^#nbbSEb&UPxJ?F zJ^$L1SmH|h5<5<(?^<1BVEpr~QhTXQU7HCKY2q%L4pPam_;Ej~p$Njg5^+veVm8M>mG`R{g_ym1_wPrvj%Pb>vB3W5JWKWzU3zn*H~tLHf$t_dnqW3THm^HUG}@d*}Sob|NQz= zmF%u|1b=#JX9EBXL?;Twfm?{>Z72 zQL_4;A=O6Cb=!+u3BOXKldJqW&)UvjP*6}j7hLo0+XY^OpS%Bd=hm;Uq^sl@H=o)^ z09H>XnBNyyCS-;0gK<9klIuFiEH3ztz^J+=C#PLh^gB8_`t#%OT-&yJynXi$K%a|n zbPn}JM1hHx_M~=ih8j7nY#?+sP7QnqaoG(YBnGxu2Y3`$AcwzvsCRuMK`7{+66{=l(cp|Edhr){c%%QzN4rmNH~_bLO)G zM*91ItXd@jw!R(CJ-&x3K6oO}KK@BmN3?I$j_5sjpv(6GU_(5qYiQm7j&a@Nd|3aa9|-nLd?21jxJQ0Q zL?P~yM|btaUg|{*cU^^T9aOT653d!9T?-QyO}?8RQc`jPkfuHR_C5I7N4dp&`EcdQ zN*c0NfxG|Q4*D}ad_3C^%ue^170nN7uY3LHTb>|(+2PKC;lcc30->GHG-6jm$9g|3 z`LNRBK_sW+OD^|4UatE2@kv5=j#cE{TxFRs9={ZIvEw+QkVxu5&Hb>uOpnK6tnoe` z)RT9G-mI_89;BqKY&L3zR3yHc6D|)pbOjgB<9Njd5={1tIwSPqFbmcB{UZjqIMFQ! zUc`h+%2Rn9K6Ul$Uq=w8Bc7zt&5yTwLX0=7?dL`us+w&PdF%Mlh&3yUX|BnRtYgB* zo;F0WL)T!~`sqBXifT#%v63^k6#AsGQc7n&ySX-HhGb$kxew5m>o-@iZr`;l4SXu( zOOmV5#k3}CmQiatZTzskO?5+lDo+s*ms}pRMm{EC8)AqsDtO~%6=e4MHy(`q!&dWwcl6}|}x*`~rHNX32Rl8(Yb z$Uh;DK~Z-(Mfnib?IYIKcf`e+%)pf|dK-F_Bc9BA|NgFQd!ILNZV@IXC$)CF#Vf|K zDfKvcnih6Tv1~mh7QuV%?JstGox1wu3BPU`S)5&_H4tD!)w|>o$tW%!uK$2T^$T$> zA0<)#_XK;~t)!M+$TM^uD3?Rq5UhG?I`esIo)Nwa(W}$_{Y+{Uxoa0AZ+VRnjnB$z15mHNMk~j5e z!BrJp%rt9KW|IpEh)T1DFq`Xc&cXC{V{(RTH|sy%bKM5RD2huG4EgusccZui+@`WN z_$%5g=VQnlcklXw)5fYS;?XMxuj}Fo8fZ)N6a15g=;bo6aPfzKnwVy>$2_R| zrP{(1dN<^n4G&*Y>ceI`W>lMK#RQCJlkGoK9@tPE$ZJh0w&gJNVo0Bt7~niVZ>jLs z&*J3ot&zpa+`Kip!!W{;0Ov)MqI^TzBgVA{9~@j{3b{|v=}WACTWSHXbo4b!XT;UY zr0I}2ENkly*MxbirRWU0vtV9L=Da1&rOEf6=f1Rj;_agVSw-r+xTlu+unu|i@<~eX zxKu+}JxAm-|8!i7wp3@(ImUXC@$PW+85Zg6VOPoYcn3cEV(E zJCUj;LGc0pt)!g&@yaYWD)+6iR+)buQsOGtcB0CE1{VIn#9&EA!l8<~7Kl{p;^haf zJ2kqKP51qlU=nt4fA)+N`pJ7`HG=r-ey80}rumAc7eF{}qKaNWp7>$I&k2|Yc% z!L~csDxyZLZz?NiS=p7&N)j)pKYv2zDje&))dK*cPikRLE!G(djZsTAr18>7u6u!$oE2Rm^iq_DU)yGh5Kb!a$O-@ad7Ao9M5~x=oam!J+YcV{ zTdwyD8@f04XiGjl*0}8(Yk5vR#O~B!AC~(ZKNlRXZ-8+S++ow#-#q&OoyW0rC&yw^ zE13@;IpY2GEBoQ6)n=p9=aAS@;mW)pPG3gC!(TSxqoXQpvLBENuU|HQ9dn|d4=ndU z`-OCJU~CLu!)Z*FhR018r)vf5F`{BlNJwxIPwgQ^pvp1l0tYTNK5BY;bz<7=${clI z;k`XzbXNM6sg@m(Pk?ezH0tWjKHL?M_A5SjL{<=L4ciTe_JswLf)#zFuuQVCvif5s zxKS|QFfgcFy1H)hotkhpiZZHU_oX=R$%^-_KG=%4{mi`a9|_s!+OxbhO{H;uOrVZy ziHad*zRVh=w;~cpd8!?dba!>#1u08d>!sVwN?j^ZKCm*onr2hE7r>oJN4`(i4MG&RguQe$Nd+PMBbeVp4NWn23LBK}Rj#Yg%UJ z_l5>vgztfcOZYiXj^5@+T!RU>w&8Ja*$AQQ$A_Jf7Db#Xr-o2AiR_5=_a;~G;MM2s(rRC0PdS|%g zm7*ZskOdIwq^nC2S6=B@j&)p=lGw^c9a?s^9EKXLoDC0uqutm%O7=vpeA_zE*u)za zLM?J!%4=kE<0Zn8d9jMsvew;k)UFkea0u*-a(%YT=T=nN8Yrx9p6-}SUu+_8A-HT? znUjZW2MWsR!R6bK+=ZhEx&>uT%O_PV6&*yvOmt;?o;PZ3Ds*PHvz#KPI?@U_?L~e= zO|ZZr)TA9*!MXVCwgX#9*E&zo*E}OMSE{ zcNaqFiHKmCAR$j|b>jy}oS!36B=t`4aIILhb+~JEObq3LTkm#~|2b#8Od~AZO@!TE zXwyZeRq-3*Rln+^>xRuzC?{ab`>_;2_XV>Q@ll<~oMFNgtDp500!BrEQod4Ng4AfE zZd06Rn0t@k%-8(qJ?KFHNf3zp&j<3_(_!j`VP`Ngw-c&N(e=;T-e!Y`CE+46iH?o_ zGCpofQc+WLj0kZsYBBr$gVosBcsCi>WlSg$!PNY)qOLAKZap12htGB`m4oVuFGJ_7 z%F{hiPbJJ;KEIQk?y|oAk>;dA&jy1NoSg7X0N+4nIia^l0fOzLXw{YI@lY~RxJk?cN7c{w60Cw?Duss`b5Kmg>g4JQ?M*h_(?>QXa-XD(ZJ9C4g$fxl@{k-A z7JdrCJCD;3KGa!~d)NE|-#tg!tp3%pE^p-qd~TdThQ9&*=^8>dA#Onp!(-oaThzF5 zx2fq(ldf-->pN}$XZWCQA|c@R>TgQoiCxg0PGNuXBBC=cSpOC4DzJEuJ%CL%tP=VR zGb<~W)NpR>Be>iPSx$H(Bm~NqXzgcq#=U982lLC9FPlGS3KSO}!5dsOdhQS+`nDAz zw)odCDmGaLa>R<9B9YvOS6Wi(7k5&LlNjfVxZQ-ixjRksl_>K?gX1eWIF)x_jca=A z@9%|_4-}I$UOSyI+4MMEmF~Pnfu}{DME=!>2>ecB6K zbmX0SliMk)M-WoVwh_elCz}}819fjdj!4{1x4BGFF!axlrT>bb5&iysh?s%d+++U| zvISgkuIT6-RAlpsb67D)1xv&d6uB0Yv=wYv9H|K>GPUlpUHHZTpL=m_t-lw2>skGp zE1$FqSHwjgHm|w>ynq;CM>n#+6h+w0mY>v%((%V5J-5H4QnAopbiq!f=|Ro?%{*Et zmY+R)>5W0)Xj9xSLqo&mWt)NE@|X3Z!Rs$I!+Q@=Q-hna6>;6q4T3PC@-NT@H+QN11uv9h9~uOaFz4tU5Y_-ScrEk}R4 zVeWa5Mr9kmXF|dHLxIbe@3&=HG1}$#(Aj)!9Y+fL0o`)4k&$q2RIdSZBixV27w-}; zJ#Z9L`W#2a(+_byv08|Hkxt9|MB!6vb-KslefK8XD2WGe!Cm?y85NZaMlB{EE#7Ok zEX2x&Rx~HIwq=g1qku;0F`NG>CE~KT?n(H8Je-Dca`?PFVrA)ht?S5s96x3g_owOp zTH%%}>Y;N^UDxm!zk2O4T+=?6I(YujzgEk~^-a#LYuSun-&R|i)K}fuR375ln6NOR zusYLJFDfRs@#TKDND@KGH=tcy~>a-`SByqJ8tHZ-^EhkR~KXoqbW(`jX@H=i-L(o zNT@6)AA-N;#C!>yNh9@IM@2-2!KTo(}d;xhsXt{ZKdMbL`5g}jH z)IK1AutmLU`}R$vK0-PMH=OtAuo|_aSBY8OvExw{@9XP7$bUEH!DD*G+`MXQ&cCV> zlSi+svWWZMzkfg8ASHrz)l$mP=L5_KCx*ATHxYn_BNF{QQ7Pjl170mY9%1ieqf=k(!$-D%(*Va5KbV z!^p~d0!1Uv9-sH`rM!pQRojJ~-tOtUgZGDajqkg6WGXz{x9{+eFmC5rpDt!$5+y?& zK%!!|bccf&O@6_0ltx}&zGW%Ul!^IIOA6eld=N$*J3aaJnd+wWf3Dkw;o1`aC}Wo| zdj7v$w--zw(F73S?z}^b0fKYXHEri3T!E5mYCDBOck-S!FS)?v_Pk&$apM%yX8*ep z=kA(ody>m(a8pUW=40X)58`v#Nz39JQuV3yRWBGUYZ}TgH(h;{w0l7IW zLKdbubpLu7`c1Db#Gd-UOxxVWZlXu`-!I`m@7vv(fK&Kie+m9`lLR~PKR%iN{HEny zZ~m`&rh*e>W{X{hVmEl7u3{Oo@60R zv7X0t^9RG)No6jVR|DmJV`JChv7i?)z293fWs$5HQ>9MED13mOT?(zv2kRv#A%)9t z8KC}AS{f9@8`~dT{(gS`_M)@%n@62u_D#(P(LAQc1FeTl!FL?;4C(D9mCQGJAN+mF zQ25MlVM4ChZQ-XuhVWkU`-n3rBi*jv9;zdIw@D=I=OYR)w_Hby}G$!k%@hxCHc?l>`&a5o^vIu|TM2$azW>c93a z^2#f&Z4Iz1uRq$po$7wayI0_}_y&a4y=6UxT(K+OR_IOCT~tc?$ica3@p~H-Oz3=V znzY|IQY(dwh}5l*txAq;vNShhqz@x!?253H3~mw?RaIZkEX~c`U#uC!=8yfR1p2b< zCmU{7*^wMIA)+t90f2q#j?>DESAMa$e~@2F`M?ZYxH^?&Wm+$o@Y4f6aTY0leo@E4 zUjGgp0-Dt=DZLe8`D^CM02#5Pgu@M`7(F-eh19(5#8N*OiWR$&{{U$E-8-tR$>rCi zyY>&M#3@xDL3`jJPVB$kh4mHk0~Hk&%e8Z@xp%A+m2HW3;)6^xG%f&;{2mWxFSkS1 zI^K z4veI5$DrEBA^p`P6brj$A2pTuI^~0#%nyDR`J_0TA~~J+LA^S64O|z}`uc3LVh^u_ z1*7G-K`y3MF#AI`X{#(Ng}!~ES?)Gxb$uBcs^;U1j}#OYw=CN+tgJ_|WMw24p^<%* zD8x{ENEdq1>K=k|TC7BE3Wd;hEp9G-(fM_%Z*;H%igiPxkQU8&hnW@NMUpQ;4tFvk zA>-i+C4Fn^<+vsiw6~sX|d2jPApQ^ zTD;JnUSD9px^4vQ;cR!9DaT=&YsmdvT=XWICK2+QKbDAwm`yFJl@-fc&K{>+9tvi+ zH*W&3cn;SFikb4`)&bl{G3F-r4|=Fq4d(o!tar?{vRIH>kNk-VJWy{v{dJ@H5|@x}YmN6kQ_4&$md58ZW30 zJX>l{U%LTQ*0q`b(&SFQWGx>c*PzGx=Cr}a6vmXSzP1PZ#fV+rClizHI|I;%-gZw{ zw;0$b+az;V<0k-i#tt zI)cud#~^b;b8GHz?vzSv&;@NS1d+0zIpgi_{+RB3a(yhhk<-$qQkqNT?S$DQriFKD z+1blm6}HZYCx0{>a4RV(JFv>}8t(CQ-A;XUPTMI$Ll{C)R*5ByITg!?ZUY7 zAyl5YJJpsg7{#m3k)ql|qa#w^i=r)KMv;&1MsJ(L$}B!NLxieX2;-t>o}vhzKwIII zIzHQMym=34?b+xV%Rg=3bLt@DT=}NOte~I(@#fI8rKEJd24SNh2zr=JO~)Mrii!>z z_hfkMkVsREFO3#F9}l4O9VVJTAvmXdJATxD|neNpfOXsnMN zr1ScKjjHLR(_Q`gARk0Upg)XCujz}GJ!TQ`)@hf1DTNQr1?N+gRnjmD(s19XGW6Cc zQKuJUrxr6gfjb-#uqyN6tT&*EcJqIv3`UQG-vp(2jBHYX!)816iMZ&WDL15!F^R=0 z#{G?sjy-rgiwxU~?xn2)B-?MvhI7tPG=FpwAs`9vhKw--8hs#5E97_Bn|{+5&vU}~ z8Hp&k1m0u&tXHYw>}izQ#N?0A;(}hzw1UZh} zdN(*og}vHm?0TvW9zHBAY)r^%C7@+wzCq4eGJczgU7%ikFgh(CB5b=iQ>_#MyTo{N z$Q^)(@5nH$L>uv((+@o?7u;y?&6Qcn1o3$zk%3olnxK|xTz5yyQ5L6j%KeKME}Z@7 zEOv}psxeU(pJF;O*{k%6B%ev6%VfA?SQK1W)*m1Em>Bnb2|W`U&QBF^TKxldI;I-2tZE;3LK8s1=WdEJugS!wa-F zO)=pq)o572FviZxx&w;?{cq#WFRuG1J28m)C1&4Iy7dNx7etU7_Ix>wYi(QG=!MVD z-0(IVLP)^Tbo^d#G_zD_2$;) z)O_z@l46&_92JpF1QQd8?rfV@9dT)f$mkd+Cmql!SdK(D;kfl~K(*O%uT?4NKZMvm zk9OnOv6r$$A;r+ss>j0ig6~wWP3`Hd~W*0vChS3?cBYe2i6@Ni5-pW zim`Lu)@!hG7dYBAM%mxNC3+8qPf>LzD48cjA2tkAbbF3A z3xcv@#wG8yCV>KfhJ+ro`L;cHhFmmL<5r|(X)o%ZK%?P;#V~`!PKu9`bNXbAq8|`C zMg$vTRBg~%zzd}jQMGKMa-*;yzb)@R{o>*i=vm&Xf1UanV{{X;QzCKzh0d=8B|p<8 zCn;1!qwP_rEQZhYCEe})Y&EwtL+JFlfN`XHE8=u>kx0}uMkE+Ms0&-$%8z({!1>fc z!colx{zq!FO*h}8uwrg+U!@KhK`gIn!REnGx$^ZDi(5u~z>F-B|GEuPzYn&y;Tm<9 zN`wC8SIi{49jSJ@pufe+x+1H5Gmj+S{>ey`&xhIS=A`ZZr=O{%5Q(dDckQM}EA4fM z=TF{F-6XDS>`OK_+q>=N5q1BHTl8!63*%p(Me!^XrIo~nen;j*g(zjmCr$ z;R<)PMc)}_Y#cuAS@%h`oEFGa)SRA&hN3;btWCFOv#_5Q_0hb%{4;ArA*C%i>5gr-k>hV7~FPn|?e0q!$_e1;56fBzED&Ee6X z*p4#0IOnF=?>>r`su_B;1fkFX|LXS#R!>OU!ALpQoKBCn0S;2kUMt>cA4jiHXSB~1 zqYkF11{ctJb~#-riLQKWE3ux5imw_bo#nB}n6xefKFXB~2PNfiQB8B-KHx81+VbdP zGPw^6O9YMsq^u&-*X`T#cDtpqdR&m>w@$>9aeZ&xe6be;dKzqUh*8A?d?C8AQ_X%s z^pq%zYXLJw|Lo8y!EJEHk{)7{9h&|f1s7OdhkdUxf4cDsCj@9e$4@V*L`6oPCVyAh z#H%h(3pLVP!A9%;SG&Dm57yRO@>3;VaRw85Y6C{lI>mHA_O#_ zU*T2;s98(d)ZtTinm#=;%CR1=c^zb7e)>*|a>F-y(t?i6VDq7>#vG-6{Kj3zjuZ=9 z-?***{33GQFyN)b`hNJ34%67d2~d0BdcCQ9dH^oBcbxaLTb4f?-e?TpbAgvm7V&$* z;C`!)KUuDjXOp9OMqB;hqs-W)PwTUu3kB-9`$>6+eH?q~h4l#W_ltxkee;RH`xXB) z+Z$jXQmv{OA2%YuW%0!P*!lD1ek+DYf6OiZT?=b^P)A8?NLwZY-Fsl5dw#wMft>DF zF?v>B!C5Nx8nMc4Z{}uvS?%?$ycIRmIR=JvBUcdS*(=vS7rdvzV(bqcWo#{j6JTaLz`Sgj;v^@aM z=HC+&H5n%lkr1cT#Eo=ola$Y}<1PI41DFIIu0BHgb3RI$;E+3V@q^PZkK|;i5<<_{ zc1qQ%p~bz@U!7{WDt`(KjLvJg@_$kHmH}0VS=;YM6h#RGr9q^myGv0(Ndf7WZs`tb zR6x4B1f;t`L}>&Bq`SKt&e}8Y#Pgo#+xc|(Ff$BW*zy10_qx}*uHR*S19Iu1Ioh_t zS7|&{$B{p33$W98hWH@}Y5l@NX{t zkvZs@u%X7@b<5+svgRTGDeg4_N4`T?Q)tKrT1r|N%`Wy!_P511nEUcCX|>%w$%iyr?8F_bCN za7f)wzJm)qRp=-{BQy;T^XHDU%t(jh{LTw{jgls3R7M`k8!%&9PdcAMk(fq!hL=9j zKDj)zn=Awn=(T5%$hLEMD4Z^9Gq)(UW(~LqkUSx=Ar*7DvpTPya#>DfIqtlB>#{{G zs8Ssf$07K|@6}GB4hVS`Iv(Ql+fqmgnl*!R0?8{v!s6}h^dTWB=CkN%*~ZJWV2$S= zC_mwXA-u}!f(kyPpY83tjeHe?{p1Q)Gp-9uo?0 zB;fdVg}56pq1l#a{jlcD8N0;QOwJE%Mw#LRUirDq>&;Ej*$^(2&uh{Uy?t+IDfy*Ozpw1?}m*-_vl%J z>xpuSEf0tl99h>}N?UHO9ymia(CVT4>K2oX3>;T^wp*})a&lcs33qqii8rfGI=i*I z8?&*_9f^{R${#~Dh@#g-CIZt&-iGu#U@?kC7t}BR&Z#|cUQmL{0|c(Urzba#Z8yaG z2L`IGTCAmX3@v3OCC!4nP#_QPWv1NB?5a3yoKu@8J-n`IX-~_{O{)v(6L5A5ji>OcnH55{WdLf8b?FZUL!3o zP78AA2u68i84R$E<-gUn9NW`}iM7bFHh4~vWdI@9H%jKB>kp`;QRrJ@!T2_teEpUu zWege2IJuO#p!kY*Z;#na zdJ;nas9q@o#Sm`l-D4{XY!ZG7K{6C5*g?k+$6!dEV0Bw0dP2_7IqX`K6h`@JZhale zcGO-`Atff>``#V@6|Zx*tju<;rk8;AtI)?ZX1%{pzqgP(?mzv^IN@BZ(_#R>N)-|&0 zQEyAwvZQ{(pf6KG2{{I}vq44HsAl73kp`sY$X{~(X0+jqt3*ih#MCCptcsilc)dEJ zQiXlBhYB=7b}pQ*m@b#2e9)c^J){l*T0E7l;|9ZPrbpvslwFys>0IAjK!Wj0auTt{ zQ3>e)CG)lv6v(a*y8X}#TBa}-I+s#<9njbn#V}UwdU88dR@Q8!hS-z8!3pFr7ywW2 zis6X{OeIV>{+1aN!Z|sD_F-EEaWa6No`*iUMEuEDgH9WU`{fC1B5xT)^p#q)P=$qs zS3B)Jf@?MyT4u$;-3dw1ACCJBa1 zF@GUw{RKdZ!26l+GV4Rj$?Q!?HPVk$zL(y(KunY2AKA4>1DQ z;Y3kgz;ua}n}Pk3Qnrs}{lr2DF*!MjV7i_U636sG6D`%D68%rW<3s8#)N5>GVZICD z%3izb_8;9zv7fN---AL5gcih+;t*K1x3YWb4Z{{j7>P8Vso+6^I*>zjF1=cZws6YD zTjzf96f*U*u>!qx*|r9^BL?UQaB1Q5x@K^=6n`pZgEf%FbciU9!?K%e%C%E1l8!}Z zJZ9<}HgXn>yfryEg5aV-OluqZH)}tpqZ_M9g19d_K0@ergUZ{CvyqveqUkRSc`Ua( zeWt{X`v9gOv5Ur{jYAS{^TmE7#Y}c`K>eCwzC#BI=eowPmpp|yVUsTTfxT^+rE{)D;)^!3|| zFE=d@f9HvD?Vg^5C7dt$1>L$$D*=Y0SbUlE6-$VVfl)dNaKC^<%>qL(f6?^vtOVD~ z?*d$nq5(pVci$yO$HcsZ*r-Tav{GGv{qfERGJY)x2_~YXj4=ryu>K0}UHOxPsxVm~ z=_-5;Bx}0R8uVC&*T>XvQr4Bg;tR9G&%{E3d=6VEGL|qNT&|6406oXVM;heKv{(p z92(~s50Slx^c7Is;hNUN*{St7jBri3PP|3*vvB^<^!$7qG!v1e_pLhWk>wwaFZA9< z*zwZQ-2pNM2xcW$i+4s9cBwejCq2p^4Udca znD+Geba5s;QbMKNnx{i!gPg0?STX1tqwo8bho)G$G$j6F(ugcO=xZb;K z8l~!2;zUQ#1ToAI@qxC9B?gMNS9FG_^O}a_8`Qamg4eWDiI2)uIld<&mb3^{8 zMSF*luC4|gOLR=*9Y*U5fN`P!D>Eu0J| zO$AX2Q{(aHbTRko#9RdwiJExy2h^a*-rnCY6D`acG@(@}IR2SS4>#pU@aU5JVW+yC zal&-_-h&671sYTD0pbUuSQ-U!ITLj2m!0~CYxyhZ9t&%xuM>}Gf9goUjJ>&eC$n?d z;}l&rF_;E~iQ3b57nxk0a%-0^oLRPZDQ+AWmJwhE_NuqVyU(Ew?ndV2C}fkRi-s)i zE@(lcgs{|5=BmXG8nS|L=&Z6zcWiX#ppuF0*_orx@7$cK9CtmlbG0qne}nX?5Cjn{ zQ41V}knlaf@RKqg{r0eYm0f$>r~UX0_X$7$wW8|{`I;9rFz-oI*&Bj9=;ZpTjh3FC zV2NmV_b}NX&tNj}4cbVcog)Q5me$qlyI0~LuaVYi1$lZt0m&73jwPWhvWH9m9sZT4 zXPRZrN8xiy>vJ3z176C(`#!un{cleXNF^IDy;mkGg%WG_K^fBt(|Btb$N`t_L!l+f zZ5;Y0?3-B7Y-7O>0+Bwm!T9yH3IP`n$!x|6^pF5YD?{NrGGwvQ_Kbkbq;0+Wj^?re z>Ajtp9u1c_jXLrK947&AI0|L*5M*U#h5oIFarHf!$)%$$fubIWb?0|Hh9i2AYQ8(= z!O=l?OBm&y0GU@QU_u>2tBr;QQ)eM}5cD~*;2?+NFdg&>S9}k^&<3;WCoC+XupX4G zhL;~Zy3Flmh*cdgy~*P+8>@lJO^MfGR*$a(|5{3COl!^0emiBYY%db9MJ9#ZFEm}A{x7)RL zeR^g_TKFA5V7j5mFgzOOgR=N6erZ6|5IV~c3nYU|0l+E<&ipaZImcD`Q24waqy&x& zNf%H_4r6O+X_=WNXN`{l8RQ{tgUHVPOd8EJ_AP}ogm>R7Rg=v=TNv*;qz$YX4VK(M z72K4_{P+;eGC1^Kk^w=A62;Qxa~wHhMn%DnBPJ#`uC9U7d9(~z4ba?Zef!nv*1ADfgDfS&r$y%I19GFA)Gd2gR4msUcnbp`VoBRIA)0sZC%-$tLegQ3P{fGl@`zSJ4>uv@52WHC!5w|^l&Fw)x{suoh$vK^>GYM zdx}ByI|KmrZ1(3Apxz8iAN~ucOGu#K-Kf& zGjPNUg)R5YKEH3J48)$-JYk+yYO`ejmev>aD|nb_U;%#(4SghvnTIF#6E2hNocHR` z<7v4Mlr)dtu!~_6k!e!i@U_wvD>Z9RBf?}+$g9KrGG~x~_evD-MUZtfqxJRiaN*SS zX!#maXld5=46^2e5a3)Ffy(3krTXSXeJ{`9>>f+VBH~L|7uFy(*{lJ)xjxp1_r*z|87}n(Z zY3u>ybKpESfg{3T9YmpQY=)9B2wTV;gB~+)nI-ZJCdjhWSWtL81%>oXN_8WRG2bHASaVw zwLeVpXX|@+6dBE^i{C&~T`G6q9fi48|5n`-;ef{Qe_8CD9C7TDe(lI8+4+-G-O_m* z{8-SMd#;%8W9en_8Gmi>@mIF9<7G#TQZ`=fuAVMh?oz(ek4GP@F^Zm8(>|T1ZAV-S zy<5=t{MtR6aBre7gkI0up1&gCo6sot@L8dJ?SDOtYSgJQDf=sFj`YIVpn^u@>0qV1 zkRRe%n{rg9$9c9#lris^UPSp8+Q`;XI1bihv$qPwgIV2b`^)6m#;v=pyN>)H0|LGn z6VN`T?{B!!+P4ch_B+|iP9Dgv4KHJ``b3+T%f7#w_mG%4^GfB|KBYOQx>^NWXg_7M z>f-FQbxxki_+n&)2YBDPP@m%nG)#}nSH8ZjAA^YVWz$7X`qC(QZQP%RbC9}z z6}huyI_GCc%Sbzv@!H{QLjxJJCe`mT`mrI9g%6idS{fhqyo`)gdP-#w^Yh0~lc~!H z&f%PA9&c4tG6j;UO;5|H_ouzF+cqqQ{;m}&IXQ`>e406baU=KJxBef250Fp(n!sC1 z%8lQsM^|}^Jx&_Ga`P6xywLsm!$jh{;t6BFIs1Fc3A_=!)eX%q0RL25_$=EJT0wwXs9<_ z9N6;<<9LdTz47^0T$4{L$MUgFFuf8X=Fhk7x)GvSzU$sL|RqI z&}!!&Teqt%OZk^S5DXfZ!RPLcqQQ8n(VDJ`S;nhNSDY$BXCvt!?D*N{ZajSWkdD!! zJ}K_f=RT+98xfIA@Ljlk3=Zxu^x(^X@%;lux8o`HY3-Or`0!ZTjlIP-!A1o)!DRP~ zcxrf0v$2O;*OcVlb2##pR?T$NH|uP|4!30bn!S0#GDIwy`bM!YmF(;e;QgahsxO46 zARU9#c?pdfR>` zNAWsuQY~L9{=r+N;T;=DsJft`WnciXKxbrfbYVciEvW?V=QnMv?WUiZL?_?zv<#VC z`LsMntxQ`&{)*@CcwhS!lOd~D1XDsKlS2jl-*K{b7RW2R$?Mcew)1D7TH)>6Z@rf_ zVsgLYFPcH^X|nozr;xruv6|JLWy{Rf%KxeE1%QZH4 zc9uL}rMKd(oLnv@sb#;{4a*xpRYk96m?epYE+e5Kfhxo%CEwaA7U9M$#=yWJeyo>A zies)(_2 z3JNsA#|Q7@SXf2g$i`602UG>+H(%{`*2=Bs3`Q#Q0W8(Iv`1&X{Wvqz(0C_I-^9r1 zjk9wWKLbODU`)E79~uW|l7jms+2Ef9MS1yf800VPos=2&Y7YMS(`FmTYqR+I{)0-> zp7E1>E`=V$EJJ-_jL-3gWBFH5s^`L=>`APJe}O~%FmracMYZ@{rv1j{eupdfGokl` z3bMKR`SQ{hN#f$-3UYFfVyvxg2LC!L!r~|{(fj7@+lZgR#OW>m5@plfzM~EeEGuh; z0;A=7OsqXeCb}w7v$MmdV5s}x*C?NDVmsXwO~CH>1%Q2ptE(mqnl-}URJ~2XhI~(< z`ORVS=V@y&em)ac#_EBJ&8TfP`hEq#u4s9jl2!Q`2n1cLq!V*x)z6(7G)w9_St34I z3i!Xk@_!MlGLoBgD3*RUideS(nVLB0cLuUe z6*M&ReSGpiu%y*%79P4bC>i#0(mZ(rVZ$T?Vr!S*s*VqKPh43gfhnQT(cQUKqH*h1 zWV83pyyBg2*4Dg3-munR8W0*0;jm`>L)(BS{x5iA!t>xxcr9ma z91Ej!cD=U2ni;<}yQ<@W#cmIIt*or}5_%3Uj*R{cfV+tCrVb#CPV3F8V%N_jasDu#rF z@V~N8PtjmU$|?J%PMN6>VV7_#1OXTxahhs2E0^5TVVRld@j1da77|ew zSf7&e2Kih1X9wX1{1_P6C-s2(>+$t3yX`DCz>=24Dw*dQ&p|&=;|5* z?+3dDY^lM)L)6I>tNE4DXrtE|I>W}?7`N-+z%5<2KV-g>IMbq_;ie8tP&$0zA9_=Z zie5!3C=fg#%$s3hIql5$7&K*@s!GXkjT+n7eACE5gvG&C8<`ZE3P;KIxgDe7O&NUx zKe(8g)cS{hP;@7;$&iF(tDQak4v_CnZ$pAry1~-`3oS_#CB<4+=ak07_J<#@dv1<* zSqjBI@4)RDi{?{%WM=V-)q==jU{!k3@)gdbd0H}^JWa} zE|t^$(q7wcjw#vQNdXYt53gBvHOQsfgS$g{FhNj+{O)gT5i#u0-lgc?rH#|p=q7fl z*etTn5~tf}6Yt`(%*Mj>ZwTS?5CxMYW=jr0q49iJCvH)M(6IJ&ENet{qk0@|@>46? zbO$rW3+ZfmcZhW_UVqPhttC|b_=il9opRLRN_FoFyFOl)tDsgP#T3fKf-P@tWCqs0 zSpLE9PKl43?OWHTaHIz!bTl%|H1p(RQiKJJ42VAA>D6 zglV6OHS!K2_Ft%<&&k=9H}3Cyt-o8_TA4aLa~YcMEWWlc&{@JbN?Bt*mNR)T->%Sh4Ks@FTMR8bdC9)C-S0+FJW zuUW4+kR^~>_6J8r@hE9I$0#IA`juGkM0~? zSTi3}eE$4{0n|5f@e@jqh5!9{#4Q=*FNdd)uYS;=Sg1nR!d$yUh`hVWLhk(MVc&`2 zWuYhcGPO?)>-^{ce%7e7Z}*>X5r~wfSO5L(V+=40{O9W#-sAr~RRqG58QzHh`qJ^d z{$JlAzJo8}zrK7>?*G?!h+C)>|NZS>X^_&!b#Xr(U9dU+-A#L;<1tIFY*8AgkS|`M<#6#~>JX8BtW%{_P zRz6|&jfpZ%zc65f-ek0reYqY${iF2svRH>q^z85^X6Di!|Jl0U{AOZ%Pw+U~lKB6$ zocz-=v4i%%J2)`m!}*`C3Z%&bkZ%90v4aaftp8!b_@^Vo@_#pzOd>#S_W$;t)S>*Z z9ur5@|H~`$|DPxRwY2uy$jg&(tByO9k@T0Bd`t>dExnD|q&+6823xmNs;ZsJdfvH- z@PbJZdZJ{td1O9qXorWqT?Ti+;*uGi|MWJ@~Y#iV}D{>A>OHB2WV?WwV5Z!aofj-M(O(4(WH<6-zIm4*q| zJ%iK|W5`!cxqJV9+{EBLXa{0jTUn*r;xPj7Rh|)y9>zTK)eq!eX zlRw{NgZx8C!o$LLvk%U-3TA%9GttoO^oPeR-}I!6*UTWKnkYIZJy!~uFMNWSE@WTK z_Q+wwcfS<4c&`Y)TP2j-v{rHRHsq&x<2DX}OeC>vh0kv;w1q;4p0@Sr05iKTC<^@o}6es`}jz&Oonv2HZ%%Ipnf3gLVN4oo%p9NgPs2xPW^BuDU+8op^7iwCUnPIsjX zN(`GRFXTdYZCAteX9hqwU{GXn8>LwNQ{X#dCJHnX0)n3@WG&SWOw*(zXtRyoBkh5{ zcNXQUtwm-Z+<$LxJqu|F`>PYlKG)F^xWWNku^sq z0)~nzHI68dzxRZP2c3`4rGal-Kt8J{MK1H>H?=*LEPW4$>Bpef(3$b#sZ(pzj57x6 zKx6S7kMp`?+>X7yJ<@GqPg;S31MOL`*knn^w-zUV+jDkwEYi84-CgPwv$np?n<&tD zZXH#8`5^GqCoo}mMACJRkFdc!=~*EcxG=b_W^vr-sPqVoMk=J}ex9AlR+})4Rlx+V z*sNU`nY1{aN)CMP$6D)u-^E)LD}zgO@!a>*@ktOhPx!MNJEx_UrMi>jDt`(z!jF%y zgVgF421c4%$Rxkh?sd=G3sX8PVK;{Sb?FVds2-4uo?0!gSK~%)0Eg&kYUAhRWRj{K zL0#D6DF`VmGF!O@G872p%2(`!*5wE!7Xfb%hD=^grAP@2j7dP0zq{u9=`-Gn7zA4) zwknN!K=%zY0FV`cti*W0aOa}A1z3PJ_6PcZzdXk$xXA8}ftmYe`y1+f#g_G0wy#=^ z!RDlJWJsU#Y`%_$hVtf{k*uIvu@78bpqY}zi4iaqbUAX1HXiea#xN*4vY=A~PNVhp zb;omV9~5uD-Rb9#5huHg`rUC4=iTYJAQ`w^wHx~L$Zl)Y20q+II{nuH-)hs>cSLRh z9oweK39XrhNTKdJ=lUcA2Gx5H_U5lnwC-OS$cD}&3srd>acfJv#cFORxpU_kBNsjC ztC9@DiVVU_y631WH?Z|8bkDO1t}^Z*t$MQ$of<{hkaiw8j+(HW+Yi4AxI)@uFsEFi z>!6wFpa3dm*+|;R<1!`e3X8FJf5F7#sf!&;QrhonK&iwlmk(PUs!^F z3KYUYFWH}Eep7IR4QS4WS<ePw$)`U6q< zuD?rebjNcn0Iag@eK4E#zOtl2X})X^4DltR8M9-TTI>^#7rI6;4vdG&Scv&R1-hp= z>iYMCbOR|eIuow$ogp<#$PgW=(!u0WSI6Tv!XrJk)hkN|XPlC5*Vh@(^3?aubt-rm zqOQEQGcESSDpf`_AivR{Z%d7}@ZZObe@GW)ICDON%p{Q2{rQd=?+wDcxhbajvdm=8 zTgJZ!5|thvto||h`};Z55h?j4i-3!Ly8T8R#B;{kZ@8@pY2VV+JOOS8EdpZYz(Wuo zmX@t?CJlQ2`*=7B1b4NszL%InnuoXdi@48^n8HCQ+a7iT#3SrvtVH8IotvN%2mp$9BJs^7|- zkGjg;HZxvdg1WW8)In&pVy!LkK{lw-A5%(1AX4O7Z<{GMjvh9~uCHPJUmt-L_vWC@EiQWWH_2u#zBA?PJJR@^+pGz%*Pi;Ldp&A z9>Vt8KRt1ZffsQo+VfHW)5*JGB^YwZmKxXr1O~eykvC=NEAYx599_GH2WjQ7t1JR< z1D@MJ)aKYA2kxcruQw?vT*Wf8e~WE!g-8U@ni56^U~Ei5%lZ)rFxs54BUckB4HNaz zC6Wf9}n4$mZ&yr=_JG zpT=7PXqJ+lJtEmd@U`JoFx+38X){9up&Dd-gu8BQK`D6CT=rjb0zt8)PQ%v|gdu`px zG^s>Bh++}~sz{+Hsnoqu2GUOFev!S+eajMWO8p}*dtijkbdep4^zsArZ6F_)M)<`G zl@i+NZ8Jy)}6Open)D(OAgjet<(HIZ;y%);c9YwQ4FpPVZxEIkZ@OiCIzv#ST*rDp3<>1Dfg$*1mO4yR$+)*i($8 zV_9}aQ^6(*dAk!wQv*3KT&x$nnn{NX*eoxffQWzwHgM32=p1d=C}{%o+FWTO&2%!SNjrOj;e5N&J)2Q!@*#1oAB5f8WMlBTWjRhY?1ueCj!01j898Qdg~6&)b#25lt=gPNhmaIR7$>b6celCh>nybCCA-R%ksE9u}3)XT}vfXu>h6sR?rkAYntD_yhcn5OX{l z73j3<1H%sS%*l1)xg{CBw>Xyd3$a;di9JXailivH#3QDrZVp`RYu>=w3T3dH?@$;j zumpAhI0I)}KgkxDsYqvKO+1`cV2x*B_h__>6I6j!39fVCu9Kb7QvDEmnLhNURMZaG z;nK=Rct@Kz-oPR(p8vGne}gwHA?$k{6oT@#R1h2WJRQn@F)m_%GKEs; z(h&o!g&>sfj^orBT(yI{7%7&x4N*X#uH4@3=smX5EgCQxIanBmBLEU-^$v7#+mF9g zRUNNS63q%NgVvu&(Du6gs?!(1g4B9)L1GasN>Iy%vKdPETsikuNP=NIgw!8W_C8=N zL_U>GfVUH}$#_9b4o++CT5h$Y%<9s%#4`rKYQ}Q9MKRQ$?z!_je+K*_v*T|BoL~Im z+9wyoprl$vZ@jv^shJ~#DkP3b4RcmGZj_xAk$Y~aHi0&3 zyjlQ6_KzNY@_qLYjSV)bKncTx_g5%DCdO8+Xk$eHqX@-v>l{2LzG0LS(*p$z@Omn? zEI{4wbTmWyMwfP5-53E3AnY3m5*1BdjW4wGOS)*1$sV^hP{#( z4SnK^67Z?g{c3N%U8<=bfjpJeIkJFFJeJQv5aozeuV#~6^d$AEHmgcy4p6L}o$te5 z0fHZaS~XO%g3C*P^H|VP!p4~+mkC1q$xiSJBb_H^u`eGnX;{!n};F)ikgL6{0RsJPIkqveGlHI!kfaOxFMy4(ae^zR^+ z2h$0Uu5D&l2aX=hkpT281X$TSbD^l}HLj5BEe3Dr-^wLASTH{WsLy4{7S2+F8~Mf; za0iNEZ5V=YH-THn{BtP+>OrLR4ve+#{MMcj@=@&GLbceX(Uvs9apF1vRQ~fDmWAhz zj*iT_i)M%G6TP{EYsUX*Y;g2WO8e1;Zr z3)j58a+Hht5%)qvW25p1M`3&J64T&=1zrl6-cFV78XZv#FM2iI%%7tnAlNnQ)y#Le ztgQh{o+X`pK&z+}fRQwraKrRxqOm;n=&Ad|*~Zx)z0^C&%=q5T&*BwHnRRBl>+0&6 z34{8(`CkcZ)1ZxJxV&kUas1ctxfet$W9hhDkNC$z=CKlKG4y9ozqaWIJZ6Ef!YNQAJO zjoABPYiPK1=7NZG=w09?+nc|^x#8kMeD7Z9XOg*I!%keA`QPra-0&s_!?{i8ESRZe z{L#m>BAp3w_#BW$m&`j|wyr&TVpmYx+rw6E4K?9mwt@BiD?u5;%jQ$!82)zBn>TNs zQ%j;1xj~LPFAA(M&F$?poz`au7dYYVUB$`4HW27EdRCS$wH9SIQ8B<7BA)pU(39(v zrPu(u2k++bd^#h1cB6|bJ0m8rheWM}A3gd+Db?Zz+yyv%I!Gs;`ih0*%j}u5uVS2y zwD=EYj}j9Rvqs34n>-vnb8R~$)`A%ssxqo>M}vl6Q;Ig2IiAX<;qg^iW?5b}gEpV% zMSZ{Jz|N&RR9!4X9#5#k(YYKPz&uSu(_4ZMYRxZ@Vsv?a0(t1*G1;8B^QyE!yWW6S zz3T3!g@t;fyZ4Pd(bu#sE&U%+2<(ZW6Q}psd5tq6klkkG;6c>XMQR) zuK_*us~O>nhI(%=uR!touaY3S1XX0@6y1x)P=wCRJs&?mR_h;cCiAsIN+I5R`pmn+ z8LY@maGV8nLn;xkt&gSs7D`isb3(t-ALB*X_lE*faG7I2U%x)S1aeDQwA`V!%3-Q# zIGGT;zcKL%zclGUrP{Dp(0Zv|s`=UtqrN))xu3xX5I)wg5petqWa(2ep>R)r3JlM_ zNAM^L*1K6!n}t<{F?ly=GJ?3aiknX!w1%cr-INl{2K%=Vw;f%88F{0w{^VpSeE+hH z8W{giDCQr-4i2rNJ$Myznc|dS0h5a7L{@KrwBZeda9MSY{GePgtz5r>rr#@mCsZ2v zGqLP_$@1AAI(v)eh?wr2rRS>VaZVd$_W07?;!*U^z0nTfJfE{MnbrD^Z2S4$Um;gr z6~ufjcv5wcI1mON+_6R>&6XTUp=P`18TaJspqP2=Cs# zdl)3FUiuQaB{shv0ao|DgAF+(XF~rdZFmjrM0#FV%5Zo_8gh}?0_)6=AXHCKtK2MF z>ntoCM}x?g+3&4W;Glc}@hpq|nPng zmJZzA+7m*-=kNb0_H0@n{jQc5gQTbD>InxcXEsY77ZSs$Q0dXDm3#q6IViZ37B%BJ zCoa!yHTt8~%3kR#_9XRe@Hs$ScD?y&i$6Ax%OMfS{=sw>$nRuby0TX;nY1heuuT@c z&niQCO1+!N7ERvpx5^uLpkxA&k8M1-Ji9&?k5q0UNRpt`pOz}klEPo(x_z}1ZZ>l7 z&uZDQ8C?6HM$WHlvc#pOXF%oAm00-&5|mOvdIYu$Z?wZ~z+#^}?q@fuN8O!0ph7{+ zM((|LFzk#9n(vIpt=H`B&!mISTkZJz1L3;RNw-Q8-`kHU7#WjfT$zSyKA(31NoNq5zU;uVZQXs~NQxKC)S$xnCEO9J@INM7`y)C3)3Ki`US@1UCYy z2mj&sUYE;v$0y6S<*JyE?o>}XcXDnHZHxi&df?CjLb~HlRq3DotngT?qoLV9Eo0i^ zbc}`+by=X;LV|R6Tc3NFs!G7HB;3WZEavx@JnEa)dG^e?3SJ0*3reBKpx*ta z;4V`<2!Ba=W8n19VmNsXi-doF=??l1+G}a|!^eIj332$MvTZZ7*E|8y*^?vVOz>G0 zk^`gw>K5`s7?PKH6L~9^V2`Cav&UcT*7~ez34L{y+OVDNB4T;a-JhKNpi%P<>obaQ z5t=Q3lVwW-xs{z@;fduoj$_bpboAlql!XAIe*P7;ukXqBX^cs=3D}EB$x>6-$FArp zonOABfSBwb+RT3oYTd8q2%z{(Q!Y+3jv6*8cl}e&*2mHZ0`Xsm5gUE!Ld8x4&XcBZ zk~Os4yGLx6R>sQ-IdTf$~8e)Hw?F)5rF&Nl`PxcJ<+&e3bljJyKCD*Su5z+FsHh{~8!`7b8*wB5% zHq&sW15=It$`2^xPVA3J#w_b+7vj70@;A~8+cgs?Ff`D_eS9tgX=u7H$QmwV1 zLwlW(u@!l)xcKwugetuK5eDKe@!s`Mt)mA&2M~xoEk`3N5I57SRosQmcC`G6lJ;~& zej5bqiZ8R#-V%FBuHxE&$b;+4gT6JN`qlAQDg)mrRJkIBEfk!?)kw zpcoo1(I;md?HEe{LE&$W-AzKlDZMkyTbSs;LDIG~G>ji^ZjNL4F*^7GAKI~T9OmDl zZ}EFAIdkGl@PJ4GWd4M>zo<#=^fziSGnVCxlIoWn8%vh8vTI>+mm60$^onxi*FbNLmL6KzaO%g@a2y6r+r0%`HGH9-5KdD2 zFN6)kbyUbR01QNs;p4jJ=dZ<`qlQ>$X#wqO1Ry%?wK+BjON9f5_3V5OZj@Sqx{85x zve#4zyU*nORGEjYE8GO!3A}J^G)(&O^BtUsF3(ydljdcH3ld?B zmmuJPY45ok*N$tJjS?MXn=H76Gf@o1>8n)*^lZ zez1(7c`^*dNoThJD(U#xhn=RPh5Sjr3fFB%Fke708#&KjCw@hSJ)5?np;%$k#4XG_ zTIq4P+cmvcS3pOMzYod1OvG?j=Zc3j?we87mKo`VD zB_=^1y`@HBz9aU*RL>A2cM>kiz2||mt+4h}u&~?{ku@4$7Jw6@*>g0AR{W+`c|Sm0^T%LWHJO}{rW_$P`9IZv*%3p&y%IWJhgaPa=&7SD-=Psa;k2@{nlu$ zzC0xd;GRo7Ghu$2#9wXY;4lXkD%cKxo(5$>#}A2VUeBIGE=$QW2_^beuxJ0s7#e79 z{s7Zt7|_V)t3G7Zth=sEcr`Ju&U3l<8j4X=wEcy2#)hvP1e|7nBNjN#hYJ`;9X7he zlJ_4&4*^ka)99<3|Ac?y5Iiomz`1nHflehvCZWN=Vri!37ZfK^;QO=LniPnuG!iK_ zkWwjoPx9$e_uS+Jf`G#$K(o+r@!JRypFMwV9*{GyN}+%G9-y)8bEijZF954|yt-yN zQe`Z`IVh=DX|Be}Y&S|z|Ar2i=xka$!((? z$>100RxLXnvy0((;x!$1{Rw>glf$ukXi8;CCN)8gu{`Xtoy=IswafyoA=615@8(xV zMo-5o&1azywOIz4kxLbl-WEMd`Qh%9vxl^AzkoR<9ndSld{@YQTOn2JdO{0+7NV1x zr$8$QLa|Vl-5RGrjYCU&dngRW>xhzQzjL-7j72BBvAyvCPJT!%GP;%Au!^ z-i%EO3m+=D_|+R+BP*>NMXucIoxm>$_J_+{C9PY?Q(zI9H}awadvhjC1wG4U=k{0F zM%4MpN`^B#m5>>@!0YMKs*1_Tpg=aVp=3{0AS8u!q?4Ux7BuWtSbZI4vgfF8VU`;N zk2KieGN`iadR^gFm`!{TP0#XZ6#P44Y&=zL%2^nDSMmQ$u)*DF@4iC`rE!_m*~O75 zte8a(z3w%3X8{msX?@bK!0WL2s)vZ@Z&9D~Mgh?E1YK{Xrlz`{400KFgakTyd42cU z(T!!bqk@-LW;_6-02mESK>^fTl6j{udl{cyk9(=-LVTwqvfVMJ$ws5RJZ{nP4L*hV~>jgZ?Z&G{$pt8(?-(4 zV4Tu|AGd7#39d)&j}8UXmEj|Kh+F;zvt1O^BHLc zYA`)*h9)Mm_FkY<%8}2GG9Lx4WVQ2wgO`_=N{J&%i`c0SJ6;UG6+g7Dppxx=lcbow zkZ8SdJU42mc6US`ESFhh)EY2B_BS7&_1%l9)MkM$w`{s!y3A{`Cr@^nZXlqyI+U;R zdgi;Ya>Y5@$47L+eeXmNw_NZ3@NRl$(3{k|tN#g)zlan-LfN80tB_aX!()RBBn!}M z=nZyc&UeN~!@OC(VtuS)mr0{W+{}y!tS`qwB!SNhskx8IfL4Qa`2qzh`Pjz?;~-bD z{C26K6!nuQUSQObgU%*`-r&ws&y_foVZ#Ml*<;kC5Ag6LA-nBz?k6%nu3bJG1wa~i zii>5AH~Nu>AZWEOhKJO`4}U>6SX&}BS3?7FV8CaJ$J2r^9!F+kCr#Ae!`c6m()8F5 z10T{{^oI)y5yz)_mR45Qv(rcZT2&fwpbUi6F`K0>y}{M;p{XjJ7m!auB;-CTbHyYa z;DL?Y*#|!^NPN|(eZ*xknI`P(@bzuZHK5CZq5n1jOMu4=(g6yjUc@4i7kc5L)G{QB z?iC5n2ZZ;jE|_nVT`tVd%>{;p$gpi1kCs6h0IiMoV3R;I1J3x9&8Fu70S`)hiu78V zbaBHp4taY7U+lA`i(&pUV!wZP+cO}bja~L+xIitLgar@drKq4wKlvKeY&};zOVmTU>&pC&r13ZusYGd^S zrRYC?{+#8qcmgy|ZTK4?X@JHE9yH!xnwacE%Cq~$!ISPd4xJ+#ma9It@CM&&k}aqc=P$4@$4vBOg9VM79n(>4XsB)Il53ikBFNDSiel9aGBH-0aF;6;%^s{8a7U7kj+ghf8%z_&lVhvm2cJG(W(<~){ zB3q|Ltk`7GI*Lid=%Vo#^>!b%fZ0Sff1zwT8dBZk)Ul5-03$vxAp3sdbFgS~F0k8~ zrF@h%c@uisS&;WT^sPCkvu?TKc|)BO4UDm1D&+XvlYL+)|01=YiH37pkz&UBezWY4S9+YY<*7M3yj2dk)^MO(Uj)XgeXX}5R z0Sar-tD>!D4|abHQcfQ-@0(CQ0z6f{{*({kUF8>>NcI)fr7Zek9M1sc!RGu~2>Ti; z$}NoB@8BP)SoCJcpF+Q0yMNvo@CP3v>7vr)a$bo`BVYLS@y6r>p_jLI3d2Q03M}io zhSb{ixJw-N&(H|!(%0y^&M#0G@l_DrYxI3xTl^@kRbn22E#sA4|6b!v+7x zH0ag(m=>r*;T8Vni+22QU*EmbkpwEr%=C0r(e(BqeNP~~x3{C0i;KWe7al)74d#A2 zm+R+5ROc}I*JNU+ir?U$Jh&tytlL< zq@d{~Mun3Uz-A#m6pl$0Qc5h^^+^EwfKzNXI-{FvL!K$)K8-V+BSzh)fB^#0brqXr zzHL}5XEW}TdrQv&E(+PZVdVaIQgw7-!1s<*t+5iiFL1bxI~jmKoZ?xDtIEd**qp%6 zu8)3iEH4MNzAM%k%Xv!+QJ2Uk3M(@(`^k#H-95W9^TFSrP)FyUkJwASW3!uySlBYy zAd*7Pe*s68M|#uTfLgKzZ9rlwU+Q=U+Z@zC4$&L zoNdueMcB5OJD*KtPsRSo$iQgakU5HkmiYOv;4I?-Dt7Zr6ks448d9e{Nq^;PA@u$? zv~L~MohAq76b#C_@GzF;w{!s`I=Q?5V_+bB8M))Qn~iBDxl0=)U6}!otFCT$4m$II zh=K4smCY>}?i#VKJZkYeAn6&{n4L~(q4RKu9`N(#A@kF;xm3|~xR^*}dMkgg<;ULd z*J1;6{S3^CcyAIz2ZoAju|1hYH~FfOzcJrtzU>2y9YKONJ?&q?_~3_!#9O`NCG5>9 z0jwDC(%{x1Wvrvf`(>b!e#MVr#gEM;w4K?5^*qhWk{%t==9+LFaV{l9#Ph1b)hS1@ zUNKIv81ew&7J`0TtdDJu4G^P)LuB{}I4pi%{BgG`@ka99rw(#a5kEY}1`wXKgCUMJ!AUfa}X8Ui7iOMOSRaG3y^Y9jUoS|4ei1Ch0$mkdQe>1t3A&T&`$%^yCvt! zwP$s2Z(h5!lRr>Pj(c@WtEuoIl*UFRovK>df3@@7UDQQgPMPa}vU^P|Stp6>RAh`NB6@Ewc?BvJmG~J|EhseJ!ifeCSV|w zhLC_$eBhh$1K2&=k_66RUq+r3X%%Q8tPK-3U3J)L6qLB+|EUt5={3X=uCrR-u#4%A zq8|uG-)RJaB7~N(dAOW_X=`UMQ2wQu#9ZS)HQ4YTXVqB>NUcA6hhaj14*I^9-#X-wK3RIp=nD&VAb?2rCyS*7?wx?NWDMlh+NmC{68Phzn_H9v68I5 z!;K$}$+tjkS0Ym5+ol0oO4OIr(eB7K4$~Kj@Tr(Ea423JF(Iu0Hg(^B?Rg2E|qQ( z5b5sj+WS6RIOl!8zx&=l?j83XzkA0xYj~8iS$oA?bItk8=lKL7yEpUI>~4lw<#4;t zEA8J!V(|0tC(HeKFfTzS-x&Yp& zUF>UPv0vREl%L#U1`Vl-Fe%VRv6*PJVrF6K%r#IfIB9f0-om=>HM}~KFPTA5C`iH! zUD@|tZ+G`ex)Q`_)A^P{La8~9TRigVN^9LaYlXWMSmFR|=gK5_D&jw0<>6Jf_ba|t zw?atWr7}A=)$NdBnA;M>6x3%@I+ki#wp3PPHX^=QYN`emc>gXTAwK(6h42<#rOd^K z0^_a8O#u@bz?|0lY>iWjVDw?O4VPj(&`C&4%yk(k0a^g(%u=NQd-?u@%_Zd3&d$yS zlO1)XwC%eI3CyNDQ@D{m0|(i+BV5ZH{#+Joc~eND#=#cMy#eI(H2TS%Pb|rPS8(iK z=<-EW?Wr`I(tGJ9XGW{d6=$??D+ekpsrGHT^9|Y#ravpbwHIY4cyM5=r!`kjO-tJZ zI{B;n)glwE0wO`sMi1fxVp&(;GNM_f{1~JI1P|8HPWp4A6>C*@sOjh)#B~%=EL2FF zF0pW~WUi3A4pD9=sZ=;g(D;PirrCI0ct{#CwnnFW;7v&m35I>ysl2(yU}-4SlMn4y zVolCw-SMO{y1nYz$m`y7yTVVrkm`{ps8xW=*H2Fu8~tjJ&d5mBOb5i}grO!zvLeS& zsykgXWaYP=FN082k4$vM<}(lz4j9kxL3d==Apf)o0_FM`fwIqc-V1>z+WXTW-*74+ zAz`(@`i}kL@Mo21zSXZ@y>@L=x%!=y7UT6^g(-?hUwm=~p*6U%p4JEp-5kTI%`}j0 zWUpK;gk=@a#%@<-_h6a5diUFz}O658q8pX_n89Ild| znVoGMA|&MG<^2c?>$h!;vs#HqUz)gp5bdKE-WRW}HR#RyEHakA0*khb8xG~2#z*X+ zkDdsGu`;=Yo4eF4K|zbLh@;Q9MN=Y8jt+w;vj3YrB#Zlh=SS!Y2!t?6Fw)81Gj`gF z_Rj&^m6o1G%~}!XVTnIX_Io0xi5*F{C8U3T`3iWS4UY>&X?y=Z&jZmvzrp_XzwgK4 zN;o~1IAr@wz6Z0t((?%Sgrlb?aX!&SUPMPpzOPcc@j#qXdag;j^SL z2aAqLn4dpiPo%w%e*Y*wc4O^8C4ck4Ns!|vAK%va4Y$@%WGU5L(8X7(3h}uMIB(x08lY!U4U|-w)<0~%y zsq8S{eW%^*P3%fq>!M^AbOH1?!71vMld4REJP-w|3vkP?t?>&wZ7-WPE&a}R%@}aB zSRQ`Powny{*ptzjr_#hTm_a({0t(f#3!QEEWn_?J8^jj|No8bCfqu1DrYR1X?lSI7PaoQ?<`DNNDR)o8>&EQZ&Xu1HCovO ziOdXje*bHFC;6>r0R!pO^9XA{6I@hM)SjVE3w9Lk`uTVJD_Y9zMa;)0*$l%*Ydlwt znInwfrRUrc z#nx-*K;f8~e|Lr07@i?ipM0SxJ{rVDeDh{DKsiG+UDUm*wIJld3VO?~#8PYM5dgaT zII7Ur#_c3Sz2+W5CX%n|6W^M~Q{9Ybd^1CZJYc_*$B@|D*VkF9d`hr?V7fbwr`Tdr z3$RIT!P+}KP(`N~u*o*u8pP=2du`pTA)3$!*JKS14Jd0b&za6W%{L$g{6`ku)Yig* zkeFBvFa|iwuiH043b8$YZY#89l4sCHjaJQ#WJbonO7u22_W>#v4vO)^3Xe)i5T&zcfOL8Gs;&U%y^W|C|wM z47ui&?!%^>p`im92@tPaofFY%3Qhp8Ks&v*$NhmUbb@xF6W5efw(dan3#zwr`44np z4OK+J&>EUO#+l|-X#wl9Z??u)o(^AfBcYX0_!vJAsImI+u4{LCt=_)YDBlBV)%72{ zkh8xLfb$0${|(gACC$uEL!B}n^ohGghTa?-vhMuu2v18PlR7({)sw5l5iR5OG|Oy7 z^o#yVRo*o@)gGvNABZ2(jvDQz*P1$2CSI(Sm_MQneH_IZ75i2hN?fAFQ zYjbp4GO_79&~O5j8?GoIaDYNZgY~E8Pp{ zLj=R8ZRd=YH!Y8+DwoIvEJo<~oj!hbJqWw$d<)k32+(8x{t*c8o2{*AJoh3p>ibaS zpyINiLO`VsR6;!-P&9#a;)Md^J~1sVgr#2Fd6}qw)I{j3!TvC(XDLj8rX7ZLfkMVJ ze|>r87pA5d@zMuDL^=SL32kNurHn`Wfp!&39YOGz@Pg&Oeg1p_suGq9yFMgHB)rtj ztVzF!+lKrcKFpMuWfiipCh?k>n4omNuUDa+xAPnnxlIS*t^MB6;4wTrD(LCb4EmW^ zPXrLThP--vtUW|D+3yRJ?)i&?>5?Q^^+*TbV7WL7RSfF{u&YpQB4HE`g~U5pc{3Ck z^-4nW@F|Ds$*WzdrM}=Fdquf`MCSq0=s>t!A4K}%fLj4wIs(MA&dUCZR^dz>9Ulj8 z2jz316zSuW&UQ^HjroFb6cp3R&YmTCx`_i^r`Ywn_+MrA*VGCZhR6J*Z|ZfE7;?XZ z%BkM2x~Gg?xPHWgkH??y{%nY;3g@acJE(nxef>c*EL~@Jeer&vBq^WyKKUHC5mVmX z%gO+UX4HZpNwG~3PJSD-I3r^I%=)X8eZx~?TV;J2K+VA_@0$F%IUluO{UY%ZBR8!V<(5m0D(19E_ z^w{q-&)MJEA@RvK6V09@OSYz4UcYshB`-2txd_@82?`Ceu+$dcm{r7D3W4wDR}cv{ z@6=UFe=6QW6P*K7f*Z-KpL-(V%UWN6EoyX^Yw4nmej|YZ9Ah-&RO=Uq zu9RnGRFXgzg`8lX@sJ0s7l8U;M^67}%{2VUMW)a(md6!wT$j+qFm+hJ?-Ybe(@$>8fMP(B&nT(}2A=77kjEooV*hfX)D8zFmo< z?5GOii+dhgp7;c}^jB{~jt^*>x}6E>X$RI6EtNarBbOw{rl#D%40fPoYxLI&8IWcm zX z$a=N%17oi;IxstwNeL-w@LPa%|JRf*kzBAx0?0{KreV+MSfINn*wBVGYC379;`B`AEK93a3y=e-Z{t@&V!D57NWA ze&y)Uj@8P*yJ@lT+nU*zQ98N(?EGM<-;$$MoUV_|OsDC2$IbaT2`^niUyskWL$ANr z)j_=25G6zlBD^|Zzh40`FCaOhEy2?PlqOuOUkyVd9_uO>&XtnuRAZo8Xds3)8{>Y- z4GUGh`CtCi^6~&xx(dRT;Ul>aVWw#u5KH;dV?>M;q2NV)XyZ7@!J@|kRW>MjeeLYT zgTaA8*y*EwV&?WT03>ugHX@x+Ofy0bq|{1S`^v{aAMd9S;`ZIU#$;cEo2PHr0$ko$?ab&+OQ{lpQ!&0-Avl}$GvCpcF zLC(y=3j6pEYS{oE$f?(M0e!!WYY*tOxL_FgB_RQ7FunPXjE2ZPpZ;PCog@J^AyPm* zfb@xTb9XmEQ0xcjiBPHVN@NMS;kq*2aR~uU#`5^n7eEO^mGmwy;=4#Acu3ecDFJy! zd^k4^Ebu~jfA!_sjR*n>@$qY1#^<*pI`rWotPzv3Jc20u34k65HkZ7VQYGZhgAE57 zkH8WT%L$Ld>MIWRsD*R|A${y9AOPqa@u$CZc1fuo_Hn1b>W!s_&%xLa%v0`#WC%gn zB-V3-{)*x)at=_AZw9%(%b-wiUHJ7<(ooiwt+YC^U@J7FQM@R>BDVEMUceh`vhYPY zSpiU`A3hn~f4iWTg@E64iC|tV(0P|R0=!TLq6G-fvZSBiHu?Pta&}Kcu)eA=U})9z z=7+-Xmri_qXDMl=9vY?PlLB%00v{g; zKC=i44cwQOX5r>uIhm-+A?F#Uh=M0I zt7%V*fq~<>GLzU9%4<&pW>N^N+4iV5;Z(qP`j0owy#=PaW^@#mqzVkVQD_4Zw$P>x zS{~Nr4kbA) z)BO#Q14TtFR+tK&IzP2mK7an!6=Jybb zy!cl82s_jE=s$QsL_9Diz^4gRPCX&IaM+r!&YuHIMSW2B2nRQ@TQ9ax z3u9t$xckw?KdJGADyKW(mx&0;nbiu?ys5a{U{RnCBg0&`sdh`a=5lj@>zQu^9hs)= zzip>+pZaoz+)!@?KqajD?CT80_9^2N5}>N}IwXX$E5+rj6yrpqQJ7yZ7?|*wsQ#D{ zaUdssdP{qJW90%9(|f^U%bN!4rx5)fWuQ~6kn;4kai1;j@a*UtrPP{^4jhmH2BOmY z_eGDu)fu0VP!0&858T{xv}ZgY3UtBx6EY##Xa>9doB0vn-a8X2sLhYKicN6{gx@x) zuRrrmmi%7VT$l**vo*QPNmElR8-5p!ayn=qFz}mfQL~!s@JdSB3(GhqA-6r>Ig6cL zA@t<1-fe!^^fPUR`m(qdOt(SJ#diE2!Rhl{;?bYOmlz53oRgl`y1+Xb<>qAn7hCmy9J;V>$bn z*LX-JZKYRTB|Xxn_cV=Bwq5aJ#(vpUXDmo8IuPQ z|D7TQ*2P7W^%aT8{^NJI#9jrfc8ym8C@-&gCyYz^ z%1Q$ah@&%|iQaq9FRB%ow5@YNkdUY>Gky_U^}=`nKoaW25_0cd=vu&@_i3TaB{&TU z!!Il+t74AXv+$lAd5Y2j{@dy*Zxu^ehu2p4*brH-afE@Dw1jgb|&IU5e zz`JD(o z0K5fC)hQCmOU${TjNEKmnF;VZj~RKE73)q(tEKKRn*~*1Nwp_kW>7zVVGx3|LNl1h zO@=(QVW}9}!VDe}okc+8$+nm%a^cXbA4| z0;crJ{fP&>0s=2NXkL5QiFKtaV#N*cDcA{E0ZWJQW#FrQOL%I!vy=q}o5={>M_~eb z1O#zsf?~zf(QB&?Fyua>;fZ(Q}-^TUH3U?yu0}t@>SWO z7?&tRrvGc(kcLOU^);UR^s>5IigGFu)&YQ2nYuy^47cq zSgoH|Zv6zk^QIPh18*M| z6NunXKL^U%7Y^Fy86lnmkG@Zz=o^yP9V^?qh4|F;O-W5O42f$X_Sf}J$E#03agPqaO0K6pO(Ci+!> z5lGp!O$P5jpRXs$#Di?Nz0Luc2*f^`Gfh^Tk_OhJloC5T==u{N0J#sgq)*Fd+x$xb zg1{ohVt9Zw?pga3NsndwOnBB%YeYvy`NpzZE&g~5SRxLmUs}5xb1f5D*GL%nFMg}X zx$u2&44B0~`36rzHUL)g9cz1_{a99)(uGWLCM98QUjF>Hi2Ph_Xtv1yf<#e{jx^Q= zz~vK$3?d@{wVk1&s(Qy*%VAZIaM)OEYgKoIWjzVa&*-Wu1PtO-0wx#n2-hzxdVC2Z z7k^(k;wR^efb0ZB=8!!XseFU*rn~vGq)j=si`7%7B=lvSpCv4ODH=n@`iqHS`pnh= zSCqg2Gkb1ahu=Rv+RBwcguJM<7;kx9A9qH7=J-m6dKtN7WC^KUau+rXd47jrb-aNr zp)J&yjnk~1<7b=X;Hl}M2uL45R)`c5-P;A$j=HU(;-v-Y09=|EF?(-0)kb!f?xDqM z^pz&$&IS+Q)KIE%pIyrNt)|<3!9_P0GQX6+KL!MdC*Bx&DZ<#dTtMz86*5j%i zUKx8;Vcg}L3Yb9oNSScKAPXI7`Sjl`KxuyqhNGYJ^YeY*`FfW%Wu1we>6~)}SC*0P z^yP;=H87B=^~@!CwadW z=#@(h)KumRBUgkRHVYDFk!C{yF9xftg})V-L1sXIhJ6S*Rqy{6(!OwyxKD9aA))d1 zabm8?B?zw}j8)OJoL#5>(Mbvn&UYZDgZE)Q?Gk2beYZjYM75s7L_F_ka<{TUn+evs z)Qu%2MWCauzne$mhv6`}4N^R@GGO>yLcXat;Jix)h2U@Q!B?7;Br`r61T_M8{oN6*rKjhoNv2DtcC9YAUV50xTQ@rxSExgCmXNU4 zm!bOS6|yA7(vuG5I#q67Uea#!K9Hw`RBkM$$p3Ezc(?5Q6rYASTY7&cz(QBnkQk=Ez7hfp=5V6my!M_J6C%|$HDRy@*`{2* z6aU0~rI^?h8UoG7K4!cu@iuTvPnLo?~tTObC)b&tJX-%@jtw;n8`c z@+`b8XZMNm?TATD%e_{56Ib^@AlN!=Exs--?HjrgXSk)Yy?ZW3sike(d4Rxh3JnW> zY(PHaW(E6_(NcnL82zgsMI(m!2cz*EkncCQw4PA19|0EQ2h53{5X z)mP|-?~9yiQ9yzjot_p&JK1{_IY$Wi`@@Pa3i6bW`n%>A2Dzilgh{4*+}jMhvP?oG zya=!z0g~OoM2C;Gm#fg-GrLfzPFBbkoN2>NaRxoYF3)(Y#cg`v{K4x9Vf?wlK`^;8 zsxSf*MDPi@(|ao(N1Ef|D-SI|B0dkv)B@)hWzFu|p`laFy5Kk>4GWZX>LF-kR+XH0 zT;+v{i7W?un*fol?cN18q#ww@LA~+Va4Ne2C)gDkmeSMB&kW=W$dE;Y#6tSR9`|07 znVDdOt72-8*>wAJSaga)omI2sq>zvhXkBsv{w0FF;p+K8P2ZCz_j$2#@#};Z)w5)Q zH`UQQYn1mNb#DGq$}@M&_r1wOcZAv~syB1?V77|0)&$kEIx}V6>Y4>GSIs zFDUSsF5HIP0oZwC#h0KQVbnnaBbte8CE7-zur&nRAjIcxJO>GOc5T+aTs;vS95j#$ z5W&s(I*m@vP$cM>>x5QSVvFbmgCa>E>bjB*TF#$PsK$0M@E0&3{nQY~$>O?Cw7T38 z_fQJc*VlK`u$DZ>&f!P49#3WZQU^TB%c`p|NL+)1?q`lX(pAFStUYXdIF zsB|)u%n$Yat_DN*<0nsw%aM;p366j#GQfB{;v6XrSq^$Md1^a4{Gm#93bl}nB94q) z44S)bD%|9fU8@(Z%4i7J13r-~GL1W7ry0*S{LyYL69d`}xCutyjC1cyu$BfU_j_=~ z#ws?9NoELBMzW)6A(DFJ)0V>de4i?Iaj34N8uVsbLvZV+tb-&Diu~2yeL~)n3)M6p>RPdp-_hZx z-S1T5zAArA#Y_7q^o#*Y3v89sdT~q!h^A^=BFNd!)`KB- zTCcXv(x5c7UT3+Q9u1EP>dhNKAxyGAeMOJM_Px|*ecJpa9q(NrEb9L#x&V3ZaR}WY zlgnZJ`#MC=p!moT$Z}8x><0Vo;R`2Ep2X%#I!5C^2_SL6-VsW4*abuU16L6ItB)2D zNo@P1^R2c8v0q_=2Qf4(>-_u6DP2|LFTyQsGV<0#?`#O^aTmkeGoSql{Fk}eqj3K4&4PO_;KHrK$fQoE_VQv za}ughx@|lD%ggVPlW)&rLbG2=L2*bJ+nXoE)T&79mo?M z=+f5iG~Bki?tD%h92^J-Z`zOY&W%BpP06_862OPA$R~1|iE08@T?%r_E-n>dvF4H0 zq>Ol)Z9x%HXapsylfY&I7cD9QU8bgx8>1a@QlOR+z^0~zE$khDXMIEqmPY}eR?sRJ zM@L_*us!|oP0H6|J4+`I9ect}#&M`Kg0!C(IB>z;FRwIwgI)M+ z_P-lCbHl6Tjxo4rqPwKlFz%vT*eH$8Y#c|g6^0!oa~5Ov7c`eD=Zl+76j4_(QZ6Fp z;|HFV^^tw<({+Amn+*92-`(qadyl=XeqX(;`y&{XfHk58e`ad>RUwdzgQazljF}p5 zkf-uNMmwlG*+C*@`-_E+aAR}`O!%N2hXa-FdNdy4y+>Vu48Bk;hVPF5SuVNQ?uz$Y z_mgVuK#^5)?M!T2-ClVKeGxp1=^(k5-K_~>7MKe5W?L$|4P{?f5xgRwaJ%A|@F?(F z>G&Kz0aqH)7{mta#qQ**2<#N3kvTQpF$SDhh~ScBKhr_g9jiPA)$m-&Im6pDjURWY zL^L%I6QwPHFwrn{fF*JtIKoQF%HH(E#}BA!3WQ9B#c-7vR7*ia8NxjksTd)d=oCFr zn(M|uwp2bvfwj&zGE4Uah^b)JymE9NyiDvg-|V+PNzebH$7tlZWT zDE1MwY8J894UDFPt?}i&yBq?x3;sh_SEAuj!2l-+^Jac4^*li50ja3!>Z6T2BvzXs*AW?Y%h@Io4q@V4wV+HNcwk3D!L73+a zB6S>D92qw<=!6`+txg<2z4QTUc1dDhf`25Wq&p4T` zTuhbN<7mFO4cdH4Sn+6f&0q+Nz5~acP0cpTa_YvqLwWv5M@L7HT|7?0>J51&FpM8v z_K$;rVom|C(8SGhgQ7}tjaR%BAvWr8NZLkCe3T61lc71KF8+Ppg;2POvMQqCV zQAD9?XEL6hy?sZF3;mn7Z;R;7Bl4j;0m>#UgAlSl4}wW1*28kYf8P(w5X_YMW0?4M z?-VE;oxCjVrfa9Js!C(KG&lz7X^Z(yc7+Th{rM;#aZD-zJzfH-i`>;uI_QhZKxX^z z-^+U?K7dkS?`&OyZ2H)oG9;!p%Q-fpp;9ka;{`?CGZ6Pa1a2YdUIs&k$zQ6~Pwn(M zn=r*xZbd1d46P62By>F>mP=8%4;mR* zDl!CYPyN&=Bre2ciEPYv1Lckw`k2e}=nvG8zlR5GIle3x#jLvNSWr-)kg4RVns4v{ zyp*9(*_-_`1b_K}u%`Addl^K~j9LzxdQLTmgTj<4#sQOwoUH8U!A!X5BDF1|D~mBH zb;3_EUB1t;bWJRV6kR%K^*?0mF%oYH*5g6ATOnC#%v2x=mc+jJ2&Z2POC>n4S~%o{ z9Fh)3huDQLyoMw`^(&C)UHUMo&hx)QSoil@A}T+Cke+tvV@U|6`!4^VRPRSIYB11( zio{Y0HDGX}ndK4x^E7|X`u`A=`aiGue+!xxg#k=Cz=r!ePT-22z6)b$`95dRUa1P# zaTUq%l;ixUp1*O<$IoLS&e#ow_)8K|K>pXm3fA^rH!YKZ?rD~rOR zp$8AxTYc=Ju3H&u))w!+e@l+(^}EX#n9eJjkTv0dvX(XF763Ag$ig$eXNOW z)NMyGgeX<3C)Y(OnTe&u1|S&TKX3W)P<-@ZK?76n`YBdjJn~#VCy}YLsq!uLMQ7bx zJedBea`qEP_wwf-d{m6@NyKr5jt)7RTTJS5=cJ|Faq&OyI`|cgPhJ(lxci>`+rv6W z9z3*P;Y~$Z7Mx@Il=O*FEDR(@4^2(vi*;#|~_0?3>g8g3KlTc_l++sRG2;`Lx z&~sIbhR@X0ss=?Kl&uyRFHCwc?b;o<`JVjy>SE;od{vKB zSU&O7*;LSXoZI=-MNfb7^amm$8^e~l%-Pi@b}+2@Ya8ZxDMg6D4}V{OFYF<+^Uj1a zJw4IsEYAg-`%fM|v?2tn@T0BW9x6`%+C&j>rhjcH1gMn!+lB+`B>%SI{!f2ud{86# zx9vEnko?9Oe_2g^aq~ zCYb)=hWhOrma+4j0X9omXX?Z4)qA-4$}38#ieBehd%i@N`1i~i+X!Dt_DdKPmN+5g zkd##PR62yM$CrMmWX-oP=m(Dqr|!U|`{6>CC28GMEuVyxpFe*d>m(>_*5WyIr1~ zS!l z5#B!w(x=jXLgl=wz}RNpku**O`GFhoTTqhMP=WBE`zSt$Q@Px z@+aJVI6B(TVw%l+X2v#XKSo7mbe2S=^z@W{4hT4_qw^@BLHoz=eAC&3Q{DMZTYr0d zqgwsou8qGMai?lle*7kbmTqWpCa0rILm%v=mXxRlC@3n%gNtwd(iUdk#>;Dg#%hwf zgwHTQNjRi)`Wf&h^Q{_Q)L73kM=YS+zC>r{5}bN6`Hq(2%$d*6zqwt#9H05=aiXNA zbCCEQ@t>u(1;B`u^r1^u);N&LMMtSu+&$QhpYAiQl*gd;`7!&J8*$%0zhfBC;U;X* z;z$B40?{dAd5b5E>Q-`YZY1}W)BD&$n&Q3K84~PI39)AOq~_#U_>|jtkB*I6Pvy4|?$6@ssYBz@X3%G2Y(9)~$ScO%o|WHD{CeHKMmZkUQL3NxAE%5g{a^4Kviy!Tq42!lz3e439U_M{??CI%cU-Sz97{KuW}I6BdX}+?3Z<5 zTF~_Bc8aEfCv>W~oSa+gc48$dDF%5u(izv!dAVs?WP5sgDa_eiV>dO-MGrdc(PN*u z#kWF*4Rae; zg$@K4hBT7~%N^&cEBh0@i8-0>N1N`8!`S!W;lp;8Xy=Sf9%Sl`>gos5(p{kA5|J)> zn%1yO&9p2bErNMw%_rhx?(Q*a5(1T%X#q^Ok!l?s9W?yw7hT=lxV^Gr(23wd#sOUE zG85CQr3&Yg7e6P$w|WT4X=zhod|$N_Y;5XF2@j_srE0Gvt-SfKn^-N(66x+|;=iJV zm@cEbcbqjSDClZl@A!({<1E+OZ`BJnhJ)XT^Y~eS$9pE=i}jqaf~rAEt)6eEJH|;S z06Q}~>_yr#(L1n-IhRX4PLv1IhoXu~n1O)-`MGlmG9p92(&}KqG2UHLrde|zn)@+A z30?OpfP&e}wgZ@E25g6VjOCfpL>vul+LqQ&DQC?A67a*&z>31@F(s zcS-qGJRxQXOKe`@VJ&pEZH#WdcfU84=g8|^+rL{f(@how-+7?nNdO%`q@cY zS+X#3*$XmI`$*Q*)y(8|mNc5M%cYkpjXWR8q0hQ<`}RX6LvJw3Nt&IdQMvB#>ER)# zKO@%l^LA#m5#Y<$XWWp6W0VnAbK`w3){f_-Kf=`sM(rxS|7(gvwWgM(S#1mo9iwI1 zVY2tj#>DGrzD?IJADJ|LvdMF$FfZ9P$5&psI~!F)grN6I+&6#1*U^ByZ^8Qs{x$(EO{B_lN{$>=8a z_o=nyc3t`BZ>dhEJpA#Qg6!(GYqa@ta^~y&e|1<q$hGAmr3aU^}9$#!xoI(bUnaJDNt@MG<83tAdSH3)eq8ptGfH$dAb^+i3ne^$^|uPfzyJTh5WVZS__H-)2T6rL~_l#ZPI^Cd0+86^$- zw0?`|vL}eF09=%{uYGlTaYsL5tDe1bpIxKEi5iLSoLbpl8IzAb`6&|IC2ARkcX%&e zjM`uC^k_S5s@B@<94WGprS`65nPH(kpZ7DzmuA%0!#rvH2#Z+IYvcF#m&Yo{0{xJT zL+G&xJ5bR|mO0Vd*wv-zv^`=v1|;N_HdBoCZ0DZ`gWBq4U1yy5Xjpn&yIxRP7GdU8 zzAQr4U1;*9xn(+m_jOOEW}PN_w{UxB3Ilj_^iARIVV*?c%e>bzHOf7gDliu+wnv-n zHs^bjWe!&ugHq|&Z>pWn;*MLthq@$>$Z8Jg#TW`4YYv&H$mgN4od=P#l?i3dI%On= zKUkBXs@`TUTHsF`J)Kgr>1w2rqOadNXmh~yBJqh37Txc5q(wvo1`TpY&LFnhttDsL z4*OJ9c^;FWk5i~cRp7*T4L*^7Mvl}gV8jVt9cN_!8V73xXY<>20zHnkYm&T-!> z2wopm-e~=$p^_DR%;6qYPr^uYY#&+`glGp#`-ZsMup)T9^V#;L>aN>^5qIP_IwY@tv?E-|NBO zt-Fq^C5cnzlEaCSqFiR3A7Yc@377io%@E3OsEupBxtFxsycPV?utEG9IU+4YzfIFu z=M8#iH>n)Cl%Q~UsR+c+b=>{t@8jK$YJFjrBM5Q?UF&qL#uG7l29cp2noN4U!yRSi zBc6V0VQp%=B9i-TXo}9ye7^SyV!8DS`+K8}hs^tawcLk0nvEnR_ZkcKEmn5<3+`>N zckntwC$N)_vXrY}U!y{Gw@jTmI(Jolc$xW69IcPiz9q-+>S;2see@cHop~fLxJ=$e zNHgNZ9PNonY9Pn)u2tpJV^4jihR@+}iiaFAp6*<6H{q%-&!D5`T&^Vh;l3zZdG42V z6gYQvkXIs9bQxKyP`Iebt*8I}_vIgSV(s zwMLq}FKk6A!rNl#$Je8?qYUNmUXQlwCq{Ed;x(?kHxDkhiQSZdp7<=2Unj3p@mMEq zpWbHWq1q?{5voSJS@;ICnrk?uoH3;6Yj18p6MqgtH+xFxD#0uI&hUlk$Pp2KjlZ6k zS(8<0FGkv;<#_Q#Wbsd2)cXUq2iNV9saJJ{gL``%%8Tt=3T>s2?$5B3d}YokyP2xz z;oU`_kh${gX8idl&ePVL;M*Oi(3}WO4W!H z%8A0SIkQWb6J9u1Aj8>gU*<2|d52NOYBggwu5#Zo_eI>(K{}gsDq@5vO*PKW%8e)u zYiYLr%yL`u)j|zZzh0w#kHa&d3UArJOc0IPC@w#VNI!^M8_>f@_@5lInQjccEP==; zb}=SI@tny%n2sB&iIehs5k5(J6dItU*N3YopVA`?s9z}r=O3iM!95p1)Pp3H*K@vF zo7lLWKo#w|vXk>nb1%hNa!#OWE%|^XJvzT2?I6S9<>A$uB|Fg98upc|tcQ7go7i>Q zNyskW%ui1%Y<+2rBR02no2K+Q_J+6Gxiju84CK%arMEq*xw)oO994YZkUQulH_zNe z4#g%r8)74q!%uW@=Zt6%j4JF>1MOaxpBrQ}m_!;TV5V)WafuKoRA5u#;0T>ixvEv$}c6i59w&p!xTu2dqC zMPL7l#)Ga`DYvaDU~@m{v2ohDUDMIcX3YC%IHG(S*N(q$T$y-Ea&$Q;TL(_eRNtNz z4wVo=%ys-%t_SJP>6WZ*xdj8Xsch7;2oCO3+qs+vYLg?#0%B2x+DS2_VJWDVZk3nz zaL-mF+4kG;F+@jQA|^BF%8e5k+op$SL=zA(BwL)hXvfdUoi%OUJw0)>T=n;wJrTGA zY_4m-m-fnO0@<7;Olz-K8igCV1kOU*+2~6?;n7jq0u~t+v1y3&N+)(JNw*aFaqe#E zVZ=0XTrO|>Rp4r!8_#r!9rDN+<5lbHJ)8zmmDTr?gfh~J5K0mVIh-o0)g^LV1!BD4 z98u+UU6yUfF?d9Xs57pZ4#Gt>FOj^12276wIa;q88NYxOapB*OcHRAYG~aIP7vhPE zWJIA&*-p+X4UPOTr_L{;+=yo4W;z863G$d&)LfSq5;g@ z?$uk{i39GKLVCpGrNfZ~B_hU^m+@{i^?EjArb~(@q3jGpmg6qcDLM%0PR6Lt5rfcT z7*}3en(7#QHIK{TLRw{)j*lLj!6cnNe+`#vNx!4(Xk2o!OTnv}E0M>~B3joQ zK-lo(q!Hom>Ul=3)JnNy(yt)EIcr&yg2>wyVQhi9zMpvxx0wjn<~JI7syjs;kz-zP zII$y~D2{dKilbOO5jI2!U3HwtkI|Xk%5C93GpvZBHKHH3C5YzD6biv~XQ2g5iSGa+ z#Tb)2jO5wRB)||IEs(bHyJB{4Q$@RC#+=R#N0Z~EdE(ag_&(mP99tK8gj1EQ!h?E1 ziRyLd*+uy!qBVPZ2j{;d^9l?w(dxM?_kKb2gxy4Ro^Q>sBSk7^5N3U6Vk}lGpWt&i z4o>grZk>@ueDHd_mRWICYtu2z6igI+J#li#FzE+0t5WKS*ge&(9=8pnPw7WJP;-Hf zVX-dvB5FV!b@Nx7C*rCrLgg1@d0Q{19yuO7yH61mgbF2yyGTQBzsCn#W5WB4Q!md{ zJ>CQ8t>>CIxC#8((7FO*8&^+jQs8{D!F!K;ps0+Bh@Re2VmUa5rX@%0lShuXb;mi@ zE+enwC(XV^n+w|@lHOr@Wr%Bg^N+1P2xVVB?3crx=@#GShnG#xh{m7RIyb8qiKAOd zS+2D1)N6USO*LLN24aMp#5<{5p9(VGd*YgW#c+q|^_bJOD9;^UO4TWq)csyTB8+BF z;6+rVI7*QPHO&`$(3w@SyR?xF-BGajSj=wY66Jibs^L|jAzluA(WpfA9*npYN!(83p z`YR`yRX~)dmpGlf(kX$E!AHw2i#Q^4rzTFS2g#Lhc{sN_?(lUqd?-6-fIjmvPT9ZZ2Nd!Z>!Zsc|}wTrz-QR7{+dB);87z zkzT`Cn=LPz$xiUp8u8u}@mo7ez0O>xB3AV9NGO>s+Q*svi1jrUv8psGEk(qK1tIEv zfdh;r5n|I@3`xG<2G5ork&}ho5>kW%yy$l`SHkB|YwA9lQ=&}-5bI;mDP78F_cIr1 zx0IJYuRhG>BZJX_%o_iF1{-yuHk(7baFt(o-9^+n@7y_F-^tVsdZMF*_O`sxnI}tD zh)|!@V~B(80_mzEhVsXgxA&E)IAjs0y&iA2ZN?6ZlpvMPxBuo}X2f*QbJbRgoWsde zx`wE)?N!w|>h;^tS*RGNx^5uGtSNpCy*mi%M0draC`OQ?I8|%0Rh2`A-_X!&)>Waw z1$IQv;jQ<~@`7U@`C(gBvGg_I$_U_i8y2eL)b1?mV;*cH9d@HpvfPE1jm{LGlry@_ zt=UUhN_9bYLiOnCXKj8DC=($xdLn!gUFD62I!lGOG}DoS)y5_pOKZLtQ9T>UE3I6# zdWHgfN9R$NK~ji{44%Nq)b;+Jb(UBX3Rm6|4$rbN-YQ)jQp7)shqORa+>b$ey1>=?$WgPD{#qG+HD7S;IEsluHZkeUSpaLb6 zN2VQZgEL*nqj}2AGTk>fzmeh5z0$GCR7CBL@fpzx)d98g9;1bEARb`1;Z$1}iYlXh zUlb;C1gV1=J7Kg{7I!>`QACa^OEnd~AL)F!r&inIm2F!ayrlkI+HGRY&|SOIr2z?e z=IaU=12)G6qx77kRu$Asq#AAzN4-jSKyqg^OMS&$6kDtaBwOLZ?mqqzLza%V6=4xp zK0I(%0j*KnqGn{0$c_Q3OI4@xR&=`JCe?9+HbHfKj@D?CDa_q7RSiOidsh_)2Cl#} z-d%ZQi0KT89i}B28GRI|ZVQ=wU4>p! zxV4xT*?(tP6TXbWEDSXH_HmFLYsWpnn3{wfBS>cR`1p`>YF^UsI|o;)blsbNI+aI= z=-M1iMM+d?alPu{IklP_lyRMCB2J*CHivxLGDbvPM4WMa?nZrTdxcM>9yrs+jzJyn z`a%CPvh;Ug7k*aBq=}OaDWza_q@gXDsMWl|zR99*?$N#8OvTwAs7O=BZ}#&>9i7(|^%ynoc$ zWpbhl{F;f;#HTl-%u~*2ZSL804HX}jk5D&Dzb?x^!r>g^Dh=C`HplC^8h(9YqV zq8u(>aL|om-)5Jravb!PM3F@w6>!SxjD{;Nt^3VCe~s9tPK>H=OUmRtsz_ODyUvu- z;Kw>1KfNe?%>r|fh7GA;<9NaKe&`?tC*Z_e!}r}6Qhi&G;_fmT-Cqm`*Q+sa@hkGJ zNLH_@1J?w*Dac-7Le2)^8lC@?t}mtjljiW_3N@R1?Cb23tj~r#X!wr$c>6aGS&e1Q zkHcH)SG!Y3xeo8XDngOu*EJ0?k>jRA(hG-QN2%kZ*_rMx-eEgxAw^sa!Mob>W7oRzI7j)(hCOV?vu-&LiV9T_|flZXj_!!JJkC88a56tF;6UG(UNpu81xcx>2- zru}G71cLO{Uq7M Date: Fri, 11 Sep 2026 12:59:55 +0200 Subject: [PATCH 08/13] fix(docs): align SDK guides with server 0.9.0 SDK setup and examples used incompatible releases, stale APIs and incorrect authentication, offset and ownership guarantees. Align the language guides with verified source or edge packages. Correct runnable samples and describe transport, retry, durability and example lifecycle limits. --- content/docs/sdk/connection-strings.mdx | 96 +++++++++------ content/docs/sdk/cpp/intro.mdx | 15 ++- content/docs/sdk/csharp/examples.mdx | 47 ++++++-- content/docs/sdk/csharp/guide.mdx | 132 ++++++++++++--------- content/docs/sdk/csharp/high-level-sdk.mdx | 90 ++++++++------ content/docs/sdk/csharp/intro.mdx | 82 ++++++++----- content/docs/sdk/go/examples.mdx | 14 +-- content/docs/sdk/go/intro.mdx | 36 ++++-- content/docs/sdk/introduction.mdx | 35 +++--- content/docs/sdk/java/examples.mdx | 20 ++-- content/docs/sdk/java/intro.mdx | 52 ++++++-- content/docs/sdk/node/examples.mdx | 47 ++++---- content/docs/sdk/node/intro.mdx | 30 +++-- content/docs/sdk/php/intro.mdx | 32 ++--- content/docs/sdk/python/examples.mdx | 16 +-- content/docs/sdk/python/intro.mdx | 33 ++++-- content/docs/sdk/rust/examples.mdx | 17 ++- content/docs/sdk/rust/high-level-sdk.mdx | 76 +++++++----- content/docs/sdk/rust/intro.mdx | 24 ++-- content/docs/sdk/rust/stream-builder.mdx | 48 ++++---- 20 files changed, 568 insertions(+), 374 deletions(-) diff --git a/content/docs/sdk/connection-strings.mdx b/content/docs/sdk/connection-strings.mdx index fbc92bb080..e242ff8544 100644 --- a/content/docs/sdk/connection-strings.mdx +++ b/content/docs/sdk/connection-strings.mdx @@ -13,7 +13,9 @@ Connection strings are supported by the Rust SDK and the SDKs that wrap it (Pyth iggy[+transport]://credentials@host:port[?option=value&option=value] ``` -The host and port are **always required**. Options are appended as `key=value` pairs separated by `&`. +The host and port are **always required**. Use a hostname or IPv4 address. The parser rejects IPv6 forms such as `[::1]:8090`. Options are appended as `key=value` pairs separated by `&`. Scheme names and option keys are case-sensitive. + +The parser does not percent-decode credentials or option values. Reserved separators such as `:` and `@` cannot be escaped inside a username or password; use a PAT for credentials that contain them. Duplicate option keys use the last value. Boolean options enable only for the literal `true`; use `false` to disable them. **Unknown option keys are hard errors.** The parser rejects the whole string with `InvalidConnectionString` instead of ignoring the key, so a typo fails fast rather than silently falling back to a default. @@ -36,68 +38,76 @@ Two forms are accepted: **Username and password**, separated by a colon. Both parts must be non-empty: ``` -iggy://username:password@localhost:8090 +iggy://username:password@127.0.0.1:8090 ``` **Personal Access Token**: any credential without a colon is treated as a PAT. Server-minted tokens are plain base64 strings with no prefix, so paste the token exactly as the server returned it: ``` -iggy://@localhost:8090 +iggy://@127.0.0.1:8090 ``` **Do not prepend anything** to the token. Strings like `iggypat-...` appear only in Iggy's own test code. A real token with a prefix added will fail to log in. -The default `iggy`/`iggy` root credentials **only exist** when the server was started with `--with-default-root-credentials` (or with `IGGY_ROOT_USERNAME`/`IGGY_ROOT_PASSWORD` set) on its first boot. Otherwise the root user gets a generated password and samples using `iggy:iggy` fail with `InvalidCredentials`: +The examples use `iggy`/`iggy`. For a new local development instance, with no `IGGY_ROOT_USERNAME` or `IGGY_ROOT_PASSWORD` overrides, use: ```bash cargo run --bin iggy-server -- --fresh --with-default-root-credentials ``` +The environment takes precedence over the flag. Stored or recovered root credentials are not replaced by bootstrap settings. `--fresh` deletes this replica's local data; in a cluster it can recover committed state, including credentials, from another replica. Use this reset only for disposable development data. A new standalone server without explicit credentials generates a root password; a new cluster requires explicit root credentials. + +TCP, QUIC, and WebSocket apply connection-string credentials during `connect()`. HTTP requires an explicit login, as described below. + ## Duration values -Options typed as durations take human-readable values such as `5s`, `500ms`, or `1m`, unless a table below says the value is a plain number. +Options typed as durations take human-readable values such as `5s`, `500ms`, or `1m`, unless a table below says the value is a plain number. Heartbeat and reconnection intervals must be greater than zero; the reconnection cooldown can be zero. ## TCP options | Key | Description | Default | |-----|-------------|---------| | `tls` | Enable TLS (`true`/`false`) | `false` | -| `tls_domain` | Domain name for TLS validation | empty | -| `tls_ca_file` | Path to a CA certificate file | none | -| `reconnection_retries` | Number of reconnection attempts, or `unlimited` | `unlimited` | -| `reconnection_interval` | Duration between reconnection attempts | `1s` | -| `reestablish_after` | Duration to wait before reestablishing the connection | `5s` | +| `tls_domain` | Name for TLS validation; when empty, use the dialed hostname or IP | empty | +| `tls_ca_file` | PEM file of trusted CA certificates, replacing the built-in roots | none | +| `reconnection_retries` | Retry passes over known endpoints after the initial pass, or `unlimited` | `unlimited` | +| `reconnection_interval` | Duration between retry passes | `1s` | +| `reestablish_after` | Cooldown measured from the last connection establishment | `5s` | | `heartbeat_interval` | Duration between heartbeats | `5s` | | `nodelay` | Enable `TCP_NODELAY` (`true`/`false`) | `false` | ``` -iggy://iggy:iggy@localhost:8090?tls=true&tls_domain=example.com&reconnection_retries=5&heartbeat_interval=3s&nodelay=true +iggy://iggy:iggy@127.0.0.1:8090?tls=true&tls_domain=localhost&tls_ca_file=core/certs/iggy_ca_cert.pem&reconnection_retries=5&heartbeat_interval=3s&nodelay=true ``` +With `tls=true`, TCP always validates the server certificate. Without `tls_ca_file`, it uses the built-in Mozilla roots. An explicit CA file can trust a private CA. The example above runs from the repository root against a TLS-enabled listener using the test certificate and key in `core/certs/iggy_cert.pem` and `core/certs/iggy_key.pem`. Those certificates are for local testing; see [Security](/docs/server/security) for TLS configuration. + ## QUIC options The QUIC reconnection keys **differ from TCP**: `reconnection_max_retries` (not `reconnection_retries`) and `reconnection_reestablish_after` (not `reestablish_after`). Using the TCP names in a QUIC string is a hard error. | Key | Description | Default | |-----|-------------|---------| -| `response_buffer_size` | Response buffer size in bytes | `10000000` | -| `max_concurrent_bidi_streams` | Maximum concurrent bidirectional streams | `10000` | +| `response_buffer_size` | Maximum bytes read for one response | `10000000` | +| `max_concurrent_bidi_streams` | Maximum concurrent peer-initiated bidirectional streams | `10000` | | `datagram_send_buffer_size` | Datagram send buffer size in bytes | `100000` | -| `initial_mtu` | Initial MTU | `1200` | -| `send_window` | Send window size | `100000` | -| `receive_window` | Receive window size | `100000` | -| `keep_alive_interval` | Keep-alive interval in milliseconds (number) | `5000` | -| `max_idle_timeout` | Maximum idle timeout in milliseconds (number) | `10000` | +| `initial_mtu` | Initial UDP payload size in bytes, clamped to at least `1200` | `1200` | +| `send_window` | Send window size in bytes | `100000` | +| `receive_window` | Connection receive window size in bytes | `100000` | +| `keep_alive_interval` | Keep-alive interval in milliseconds; `0` disables it | `5000` | +| `max_idle_timeout` | Idle timeout in milliseconds; `0` leaves Quinn's `30000` ms default | `10000` | | `validate_certificate` | Validate the server certificate (`true`/`false`) | `false` | | `heartbeat_interval` | Duration between heartbeats | `5s` | | `reconnection_max_retries` | Number of reconnection attempts, or `unlimited` | `unlimited` | | `reconnection_interval` | Duration between reconnection attempts | `1s` | -| `reconnection_reestablish_after` | Duration to wait before reestablishing the connection | `5s` | +| `reconnection_reestablish_after` | Cooldown measured from the last connection establishment | `5s` | ``` -iggy+quic://iggy:iggy@localhost:8080?validate_certificate=false&reconnection_max_retries=5 +iggy+quic://iggy:iggy@127.0.0.1:8080?validate_certificate=false&reconnection_max_retries=5 ``` +QUIC certificate validation is disabled by default. When enabled, it uses the platform verifier and the connection-string configuration's server name, `localhost`. For a different certificate name, use the Rust QUIC configuration builder's `with_server_name()` method. Values above QUIC's 62-bit limit for `max_concurrent_bidi_streams`, `receive_window`, or `max_idle_timeout` are rejected with `InvalidConfiguration`. + ## WebSocket options | Key | Description | Default | @@ -105,54 +115,62 @@ iggy+quic://iggy:iggy@localhost:8080?validate_certificate=false&reconnection_max | `heartbeat_interval` | Duration between heartbeats | `5s` | | `reconnection_retries` | Number of reconnection attempts, or `unlimited` | `unlimited` | | `reconnection_interval` | Duration between reconnection attempts | `1s` | -| `reestablish_after` | Duration to wait before reestablishing the connection | `5s` | -| `read_buffer_size` | Read buffer size in bytes | transport default | -| `write_buffer_size` | Write buffer size in bytes | transport default | -| `max_write_buffer_size` | Maximum write buffer size in bytes | transport default | -| `max_message_size` | Maximum WebSocket message size in bytes | transport default | -| `max_frame_size` | Maximum WebSocket frame size in bytes | transport default | -| `accept_unmasked_frames` | Accept unmasked frames (`true`/`false`) | transport default | +| `reestablish_after` | Cooldown measured from the last connection establishment | `5s` | +| `read_buffer_size` | Read buffer size in bytes | `131072` | +| `write_buffer_size` | Target write buffer size in bytes | `131072` | +| `max_write_buffer_size` | Maximum write buffer size in bytes; must exceed `write_buffer_size` | `usize::MAX` | +| `max_message_size` | Maximum incoming WebSocket message size in bytes | `67108864` | +| `max_frame_size` | Maximum incoming frame payload size in bytes | `16777216` | +| `accept_unmasked_frames` | Has no effect on client connections | `false` | | `tls` | Enable TLS (`true`/`false`) | `false` | -| `tls_domain` | Domain name for TLS validation | empty | -| `tls_ca_file` | Path to a CA certificate file | none | +| `tls_domain` | Name for TLS validation; when empty, use the connected IP | empty | +| `tls_ca_file` | PEM file of trusted CA certificates when validation is enabled | none | | `tls_validate_certificate` | Validate the server certificate (`true`/`false`) | `false` | ``` -iggy+ws://iggy:iggy@localhost:8092?heartbeat_interval=5s&max_message_size=1048576 +iggy+ws://iggy:iggy@127.0.0.1:8092?heartbeat_interval=5s&max_message_size=1048576 ``` +WebSocket TLS does not validate the server certificate unless `tls_validate_certificate=true`. With validation enabled, `tls_ca_file` replaces the built-in Mozilla roots. Size limits cannot be removed through connection strings; use the WebSocket configuration object for an unlimited message or frame size. + ## HTTP options | Key | Description | Default | |-----|-------------|---------| | `heartbeat_interval` | Duration between heartbeats | `5s` | -| `retries` | Number of request retries | `3` | +| `retries` | Maximum retries for transient request failures | `3` | ``` -iggy+http://iggy:iggy@localhost:3000?retries=5 +iggy+http://iggy:iggy@127.0.0.1:3000?retries=5 ``` +The HTTP scheme constructs a plain `http://` API URL. Use the Rust HTTP builder's `with_api_url()` for HTTPS. HTTP parses the credential field but does not log in with it. In Rust, call `login_user()` or `login_with_personal_access_token()` explicitly before protected requests. + ## Usage +In Rust, construct clients inside a Tokio runtime, which QUIC needs even before `connect()`. The following snippet assumes an async function returning `Result<_, IggyError>` and the matching [Rust SDK setup](/docs/sdk/rust/intro). + ```rust use iggy::prelude::*; // TCP with default options -let client = IggyClient::from_connection_string("iggy://iggy:iggy@localhost:8090")?; +let client = IggyClient::from_connection_string("iggy://iggy:iggy@127.0.0.1:8090")?; // QUIC -let client = IggyClient::from_connection_string("iggy+quic://iggy:iggy@localhost:8080")?; +let client = IggyClient::from_connection_string("iggy+quic://iggy:iggy@127.0.0.1:8080")?; // WebSocket -let client = IggyClient::from_connection_string("iggy+ws://iggy:iggy@localhost:8092")?; +let client = IggyClient::from_connection_string("iggy+ws://iggy:iggy@127.0.0.1:8092")?; // HTTP -let client = IggyClient::from_connection_string("iggy+http://iggy:iggy@localhost:3000")?; +let client = IggyClient::from_connection_string("iggy+http://iggy:iggy@127.0.0.1:3000")?; +client.login_user("iggy", "iggy").await?; ``` ```python -# Python -client = IggyClient.from_connection_string("iggy://iggy:iggy@localhost:8090") +from apache_iggy import IggyClient + +client = IggyClient.from_connection_string("iggy://iggy:iggy@127.0.0.1:8090") ``` -When you need settings a connection string cannot express (a custom `Encryptor`, a `Partitioner`, and similar), start from `IggyClientBuilder::from_connection_string()` and extend the builder on top. +When you need settings a connection string cannot express (client-side encryption, a custom `Partitioner`, and similar), start from `IggyClientBuilder::from_connection_string()` and extend the builder on top. diff --git a/content/docs/sdk/cpp/intro.mdx b/content/docs/sdk/cpp/intro.mdx index cc8833344f..78b60957ae 100644 --- a/content/docs/sdk/cpp/intro.mdx +++ b/content/docs/sdk/cpp/intro.mdx @@ -9,7 +9,7 @@ This SDK isn't published to any package registry. Consume it from the monorepo a ## Building from source -The project builds with Bazel (`BUILD.bazel` / `MODULE.bazel`). The build drives the system-provided cargo toolchain to compile the Rust bridge, so concurrent runs can race. Run builds serially. +The project builds with the Bazel version pinned in `.bazelversion` (`BUILD.bazel` / `MODULE.bazel`). Its Rust bridge uses the Cargo and Rust toolchain provisioned by `rules_rust`, with a separate target directory in the Bazel output tree. Use a compatible Java runtime for Bazel; the pinned Bazel 9.2.0 requires Java 21 or newer. ```bash cd foreign/cpp @@ -26,7 +26,7 @@ bazel test //:e2e ## Quick start -The snippet below mirrors the SDK's own end-to-end tests in `foreign/cpp/tests/e2e`: +The snippet below mirrors the SDK's own end-to-end tests in `foreign/cpp/tests/e2e`. Start a [local server](/docs/introduction/getting-started) on `127.0.0.1:8090` with the `iggy`/`iggy` credentials first. The stream name must not already exist. ```cpp #include "lib.rs.h" @@ -34,6 +34,7 @@ The snippet below mirrors the SDK's own end-to-end tests in `foreign/cpp/tests/e #include #include #include +#include iggy::ffi::Identifier id(const std::string &name) { iggy::ffi::Identifier identifier; @@ -56,9 +57,7 @@ rust::Vec payload(const std::string &text) { } int main() { - // Empty string = TCP to the default local address. Connection strings - // (iggy://..., iggy+quic://..., ...) are also accepted. - iggy::ffi::Client *client = iggy::ffi::new_connection(""); + iggy::ffi::Client *client = iggy::ffi::new_connection({}); client->connect(); client->login_user("iggy", "iggy"); @@ -105,7 +104,7 @@ iggy+http://user:pass@host:port (HTTP) iggy+ws://user:pass@host:port (WebSocket) ``` -When an empty string or plain address is passed to `new_connection()`, it defaults to TCP. +`iggy::ffi::from_connection_string()` accepts these strings. `new_connection()` instead takes an `IggyClientConfig`: `{}` selects TCP at `127.0.0.1:8090` without automatic login, and its `server_address` field overrides that address. Binary clients authenticate from their connection string during `connect()`; HTTP requires an explicit `login_user()` call. See [connection strings](/docs/sdk/connection-strings) for the supported options and credential syntax. ## Helper types @@ -113,6 +112,6 @@ Beyond the generated `lib.rs.h` FFI surface, `foreign/cpp/include/iggy.hpp` ship ## Current status -The FFI bridge exposes the full command-level client: streams, topics, partitions, and segments; message sending and polling with user headers; consumer groups and consumer offsets; stats, client info, and cluster metadata; snapshots; permission updates and password changes; topic option discovery (`describe_options`); and raw binary requests (`send_binary_request`). +The FFI bridge exposes command-level operations for streams, topics, partitions, and segments; message sending and polling with user headers; consumer groups and consumer offsets; stats, client info, and cluster metadata; snapshots; user create/get/update/delete, permission updates and password changes; topic option discovery (`describe_options`); and raw binary requests (`send_binary_request`). -Not yet exposed: the high-level `Producer`/`Consumer` wrappers and user management (create/get/update/delete users, personal access tokens). Contributions are welcome. +Not yet exposed: the high-level `Producer`/`Consumer` wrappers and personal access token management. Contributions are welcome. diff --git a/content/docs/sdk/csharp/examples.mdx b/content/docs/sdk/csharp/examples.mdx index 43302389dd..34a1cf6ff7 100644 --- a/content/docs/sdk/csharp/examples.mdx +++ b/content/docs/sdk/csharp/examples.mdx @@ -5,6 +5,8 @@ description: "Producer and consumer samples for the C# SDK, built on the high-le These samples use the [High-level SDK](/docs/sdk/csharp/high-level-sdk) - the recommended way to build producers and consumers. For the low-level, per-call equivalents, see the [Guide](/docs/sdk/csharp/guide). +Use the [installation and server setup](/docs/sdk/csharp/intro) first. The snippets share their named resources: run a producer before its corresponding consumer. + ## Producer A publisher that creates the stream and topic if missing, batches sends in the background, and retries failures: @@ -18,7 +20,7 @@ using Apache.Iggy.Extensions; using Apache.Iggy.Factory; using Apache.Iggy.Messages; -var client = IggyClientFactory.CreateClient(new IggyClientConfigurator +using var client = IggyClientFactory.CreateClient(new IggyClientConfigurator { BaseAddress = "127.0.0.1:8090", Protocol = Protocol.Tcp @@ -27,7 +29,7 @@ var client = IggyClientFactory.CreateClient(new IggyClientConfigurator await client.ConnectAsync(); await client.LoginUserAsync("iggy", "iggy"); -var publisher = client.CreatePublisherBuilder( +await using var publisher = client.CreatePublisherBuilder( Identifier.String("dev"), Identifier.String("events")) .CreateStreamIfNotExists("dev") @@ -46,14 +48,13 @@ for (var i = 0; i < 100; i++) // Drain the background queue before exiting await publisher.WaitUntilAllSendsAsync(); -await publisher.DisposeAsync(); -Console.WriteLine("Sent 100 messages"); +Console.WriteLine("Queued 100 messages"); ``` ## Consumer group -A consumer that creates and joins a consumer group, commits offsets after each received message, and surfaces polling errors: +A consumer that creates and joins a consumer group, stores each offset after handling the yielded message and requesting the next one, and surfaces polling errors: ```csharp using System.Text; @@ -65,7 +66,7 @@ using Apache.Iggy.Extensions; using Apache.Iggy.Factory; using Apache.Iggy.Kinds; -var client = IggyClientFactory.CreateClient(new IggyClientConfigurator +using var client = IggyClientFactory.CreateClient(new IggyClientConfigurator { BaseAddress = "127.0.0.1:8090", Protocol = Protocol.Tcp @@ -74,7 +75,7 @@ var client = IggyClientFactory.CreateClient(new IggyClientConfigurator await client.ConnectAsync(); await client.LoginUserAsync("iggy", "iggy"); -var consumer = client.CreateConsumerBuilder( +await using var consumer = client.CreateConsumerBuilder( Identifier.String("dev"), Identifier.String("events"), Consumer.Group("event-processors")) @@ -115,7 +116,7 @@ using Apache.Iggy.Factory; using Apache.Iggy.Kinds; using Apache.Iggy.Publishers; -var client = IggyClientFactory.CreateClient(new IggyClientConfigurator +using var client = IggyClientFactory.CreateClient(new IggyClientConfigurator { BaseAddress = "127.0.0.1:8090", Protocol = Protocol.Tcp @@ -134,11 +135,10 @@ var publisherBuilder = IggyPublisherBuilder.Create( publisherBuilder.CreateStreamIfNotExists("orders"); publisherBuilder.CreateTopicIfNotExists("created"); -var publisher = publisherBuilder.Build(); +await using var publisher = publisherBuilder.Build(); await publisher.InitAsync(); await publisher.SendAsync(new OrderEvent(Guid.NewGuid(), 99.90m)); -await publisher.DisposeAsync(); // Consume them var consumerBuilder = IggyConsumerBuilder.Create( @@ -151,7 +151,7 @@ var consumerBuilder = IggyConsumerBuilder.Create( consumerBuilder.WithPollingStrategy(PollingStrategy.Next()); consumerBuilder.WithAutoCommitMode(AutoCommitMode.AfterReceive); -var consumer = consumerBuilder.Build(); +await using var consumer = consumerBuilder.Build(); await consumer.InitAsync(); await foreach (var message in consumer.ReceiveDeserializedAsync()) @@ -186,3 +186,28 @@ The [examples/csharp](https://github.com/apache/iggy/tree/master/examples/csharp - **MessageEnvelope** - envelope pattern (message type + JSON payload) over the low-level client - **MessageHeaders** - user-defined message headers - **TcpTls** - TLS-encrypted TCP connection + +Repository examples require .NET 10 and reference the local SDK. Build and run from the same checkout as the server: + +```bash +dotnet build examples/csharp/Iggy_SDK.Examples.sln --configuration Release +dotnet run --no-build --configuration Release --project examples/csharp/src/GettingStarted/Iggy_SDK.Examples.GettingStarted.Producer +dotnet run --no-build --configuration Release --project examples/csharp/src/GettingStarted/Iggy_SDK.Examples.GettingStarted.Consumer +``` + +The getting-started producer sends 50 messages to partition 0; its consumer reads five nonempty batches and then exits. Envelope and header examples also use five batches. NewSdk uses four partitions and its group consumer runs until stopped. + +For TcpTls, run from the repository root so `core/certs/iggy_ca_cert.pem` resolves. Start a separate matching server with the example TLS certificate and root credentials: + +```bash +IGGY_ROOT_USERNAME=iggy IGGY_ROOT_PASSWORD=iggy \ +IGGY_TCP_TLS_ENABLED=true \ +IGGY_TCP_TLS_CERT_FILE=core/certs/iggy_cert.pem \ +IGGY_TCP_TLS_KEY_FILE=core/certs/iggy_key.pem \ +cargo run --bin iggy-server +``` + +```bash +dotnet run --no-build --configuration Release --project examples/csharp/src/TcpTls/Iggy_SDK.Examples.TcpTls.Producer +dotnet run --no-build --configuration Release --project examples/csharp/src/TcpTls/Iggy_SDK.Examples.TcpTls.Consumer +``` diff --git a/content/docs/sdk/csharp/guide.mdx b/content/docs/sdk/csharp/guide.mdx index ac5e5d71cb..550177b7c3 100644 --- a/content/docs/sdk/csharp/guide.mdx +++ b/content/docs/sdk/csharp/guide.mdx @@ -5,33 +5,38 @@ description: "Client configuration and the full low-level IIggyClient API surfac This guide covers client configuration and the full `IIggyClient` API surface - the low-level, per-call operations. For the ergonomic producer/consumer abstractions built on top of these, see the [High-level SDK](/docs/sdk/csharp/high-level-sdk). -All examples assume you already have a connected, authenticated client (see [Creating a client](/docs/sdk/csharp/intro#creating-a-client)). +The API fragments below reuse a connected, authenticated `client` (see [Creating a client](/docs/sdk/csharp/intro#creating-a-client)) and the imports in the configuration example. Use existing streams and topics for read, update, delete, publish, and poll operations; the create examples show how to prepare them. ## Client configuration `IggyClientConfigurator` exposes the full set of connection options - buffer sizes, TLS, heartbeat, automatic reconnection with exponential backoff, and auto-login: ```csharp -var client = IggyClientFactory.CreateClient(new IggyClientConfigurator +using System.Text; +using Apache.Iggy; +using Apache.Iggy.Configuration; +using Apache.Iggy.Contracts; +using Apache.Iggy.Contracts.Auth; +using Apache.Iggy.Enums; +using Apache.Iggy.Factory; +using Apache.Iggy.Headers; +using Apache.Iggy.IggyClient; +using Apache.Iggy.Kinds; +using Apache.Iggy.Messages; +using Partitioning = Apache.Iggy.Kinds.Partitioning; + +using var client = IggyClientFactory.CreateClient(new IggyClientConfigurator { BaseAddress = "127.0.0.1:8090", Protocol = Protocol.Tcp, - // Buffer sizes (optional, default: 4096) - ReceiveBufferSize = 4096, - SendBufferSize = 4096, + // Null preserves the operating system socket defaults. + ReceiveBufferSize = null, + SendBufferSize = null, // Upper bound on a reply frame the server sends, 64 MiB by default (TCP) MaxResponseFrameSize = 64 * 1024 * 1024, - // TLS/SSL configuration - TlsSettings = new TlsSettings - { - Enabled = true, - Hostname = "iggy", - CertificatePath = "/path/to/cert" - }, - // Idle ping keeping the session alive and consumer-group assignments fresh (TCP). // Default 5 seconds; init-only HeartbeatInterval = TimeSpan.FromSeconds(5), @@ -48,8 +53,7 @@ var client = IggyClientFactory.CreateClient(new IggyClientConfigurator BackoffMultiplier = 2.0 }, - // Auto-login after connection. Reconnection needs it: without credentials - // to replay, a reconnect cannot restore the session + // Configured credentials are reused when TCP reconnects. AutoLoginSettings = AutoLoginSettings.For("iggy", "iggy") // or AutoLoginSettings.ForPersonalAccessToken("your_token") }); @@ -57,7 +61,7 @@ var client = IggyClientFactory.CreateClient(new IggyClientConfigurator await client.ConnectAsync(); ``` -With auto-login configured, the client logs in automatically once the connection is established, so you can skip the explicit `LoginUserAsync` call. +With TCP auto-login configured, the client logs in once connected. Successful manual username/password or PAT login is also remembered for reconnects. HTTP `ConnectAsync` does no network work and does not auto-login; call `LoginUserAsync` or `LoginWithPersonalAccessTokenAsync` explicitly. #### `IggyClientConfigurator` @@ -66,12 +70,12 @@ With auto-login configured, the client logs in automatically once the connection | `BaseAddress` | `string` | *required* | Server address. TCP takes `host:port` (e.g. `127.0.0.1:8090`); HTTP needs a full URI (e.g. `http://127.0.0.1:3000`) | | `Protocol` | `Protocol` | *required* | Transport: `Protocol.Tcp` or `Protocol.Http` | | `MaxResponseFrameSize` | `int` | 64 MiB | Largest reply frame accepted over TCP. A reply above the bound is refused and the connection dropped; raise it if a single response legitimately exceeds it (a large [snapshot](#snapshots) is the usual case) | -| `ReceiveBufferSize` | `int` | `4096` | Receive buffer size in bytes | -| `SendBufferSize` | `int` | `4096` | Send buffer size in bytes | +| `ReceiveBufferSize` | `int?` | `null` | TCP receive buffer override in bytes; null preserves the OS default | +| `SendBufferSize` | `int?` | `null` | TCP send buffer override in bytes; null preserves the OS default | | `HeartbeatInterval` | `TimeSpan` | `5s` | Interval between the automatic pings the TCP client sends while connected (see [Heartbeat](#heartbeat)). Init-only; must be between 1 millisecond and about 49 days | -| `TlsSettings` | `TlsSettings` | disabled | TLS/SSL configuration (see below) | -| `ReconnectionSettings` | `ReconnectionSettings` | enabled | Automatic reconnection behavior (see below) | -| `AutoLoginSettings` | `AutoLoginSettings` | disabled | Automatic login on connect | +| `TlsSettings` | `TlsSettings` | disabled | TCP TLS configuration (see below); HTTP uses its HTTPS URI and standard certificate validation | +| `ReconnectionSettings` | `ReconnectionSettings` | enabled | TCP reconnection behavior (see below) | +| `AutoLoginSettings` | `AutoLoginSettings` | disabled | TCP automatic login on connect | | `LoggerFactory` | `ILoggerFactory` | `NullLoggerFactory.Instance` | Logger factory for diagnostics (currently applied to TCP clients only) | | `MessageEncryptor` | `IMessageEncryptor?` | `null` | Client-side payload encryptor (encrypts on send, decrypts on poll - see [Message encryption](#message-encryption)) | | `AllowAutoCommitWithEncryptor` | `bool` | `false` | Allow auto-commit while an encryptor is configured | @@ -93,14 +97,14 @@ The TCP client pings the server on its own every `HeartbeatInterval` while conne | Property | Type | Default | Description | |----------|------|---------|-------------| | `Enabled` | `bool` | `true` | Enable automatic reconnection when the connection drops | -| `MaxRetries` | `int` | `0` | Maximum reconnection attempts (`0` = infinite) | +| `MaxRetries` | `int` | `0` | Maximum failed passes through the known address roster (`0` = unlimited) | | `InitialDelay` | `TimeSpan` | `5s` | Delay before the first reconnection attempt | | `MaxDelay` | `TimeSpan` | `30s` | Maximum delay between attempts | | `WaitAfterReconnect` | `TimeSpan` | `1s` | Pause after a successful reconnect (e.g. to rejoin a consumer group) | | `UseExponentialBackoff` | `bool` | `true` | Use exponential backoff for delays | | `BackoffMultiplier` | `double` | `2.0` | Multiplier for exponential backoff | -> **Warning:** Reconnection is **on by default** with **unlimited retries**. Only a failed dial is retried. A rejected certificate, bad credentials, or a missing leader is thrown right away. With the default `MaxRetries = 0` an unreachable server is retried forever, so a request that passes no `CancellationToken` waits for as long as the server stays down. Set `MaxRetries` or pass a token to bound the wait, and set `Enabled = false` to opt out of reconnection entirely. Reconnection only replays a request when [auto-login](#autologinsettings) can restore the session. A client that logged in by hand fails fast on a lost connection. +> **Warning:** TCP reconnection is **on by default** with **unlimited retries**. A failed pass tries the known roster addresses before consuming one retry. Connection and TLS handshake failures can be retried; an invalid local CA file stops after the pass, and rejected login credentials are surfaced. Pass a `CancellationToken` or set `MaxRetries` to bound connection attempts, or set `Enabled = false` to disable reconnection. Request replay also has its own deadline and requires configured or remembered login credentials. #### `AutoLoginSettings` @@ -111,18 +115,20 @@ The TCP client pings the server on its own every `HeartbeatInterval` while conne | `Password` | `string` | `""` | Password for auto-login | | `PersonalAccessToken` | `string` | `""` | Personal access token to sign in with instead of a username and password; takes precedence over them when set | -All properties are init-only: build the settings with an object initializer or the static factories `AutoLoginSettings.For(username, password)` and `AutoLoginSettings.ForPersonalAccessToken(token)`. Configuring auto-login is what enables transparent request replay after a reconnect. Without stored credentials a reconnect **cannot restore the session**, and requests fail until the client logs in again. +All properties are init-only: build the settings with an object initializer or the static factories `AutoLoginSettings.For(username, password)` and `AutoLoginSettings.ForPersonalAccessToken(token)`. TCP reconnects use configured auto-login credentials first, then credentials remembered from a successful manual login. Without either, a reconnect cannot restore the authenticated session. ## Message encryption -Set `IggyClientConfigurator.MessageEncryptor` to encrypt message payloads and user headers client-side - they are encrypted on send and decrypted on poll, so the server **only ever sees ciphertext**. The built-in `AesMessageEncryptor` uses AES-GCM (confidentiality + authenticity) and takes a 16-, 24-, or 32-byte key for AES-128/192/256: +Set `IggyClientConfigurator.MessageEncryptor` to encrypt message payloads and user headers client-side. They are encrypted on send and decrypted on poll; routing information and message metadata remain visible to the server. The built-in `AesMessageEncryptor` uses AES-GCM (confidentiality + authenticity) and takes a 16-, 24-, or 32-byte key for AES-128/192/256: ```csharp +using System.Security.Cryptography; using Apache.Iggy.Encryption; -using var encryptor = new AesMessageEncryptor(key); // 16, 24, or 32 bytes +var key = RandomNumberGenerator.GetBytes(32); +using var encryptor = new AesMessageEncryptor(key); -var client = IggyClientFactory.CreateClient(new IggyClientConfigurator +using var client = IggyClientFactory.CreateClient(new IggyClientConfigurator { BaseAddress = "127.0.0.1:8090", Protocol = Protocol.Tcp, @@ -130,18 +136,18 @@ var client = IggyClientFactory.CreateClient(new IggyClientConfigurator }); ``` -For a custom scheme, implement `IMessageEncryptor`. +Keep the same key for clients that exchange encrypted messages and for later reads; the generated key above lasts only for this example. For a custom scheme, implement `IMessageEncryptor`. Things to know: - The encryptor applies to **every message on the connection** - topics mixing encrypted and plaintext messages are not supported. - You own the encryptor and must dispose it once the client is done with it. -- With an encryptor configured, polling with `autoCommit: true` throws `InvalidOperationException` by default: the server commits the offset **before the client decrypts**, so a decryption failure would silently skip the whole batch. Poll with `autoCommit: false` and store offsets after processing, or opt in with `AllowAutoCommitWithEncryptor = true`. This guard does not affect the high-level `IggyConsumer` commit modes. +- With an encryptor configured, polling with `autoCommit: true` throws `InvalidOperationException` by default: the server commits the offset **before the client decrypts**, so a decryption failure would silently skip the whole batch. Poll with `autoCommit: false` and store offsets after processing, or opt in with `AllowAutoCommitWithEncryptor = true`. `IggyConsumer` separately rejects wire auto-commit with encryption; use `AfterReceive` or `Disabled` there. - On the high-level builders, `WithEncryptor(...)` is only valid when the builder creates its own client; for an external client, set `MessageEncryptor` on the configurator instead. ## Connection events -Subscribe to connection state changes - useful for reacting to reconnects (e.g. rejoining a consumer group): +TCP clients publish connection state changes, useful for reacting to reconnects. HTTP subscription methods are no-ops: ```csharp Func handler = async args => @@ -171,7 +177,7 @@ Server-assigned ids are **0-based**: the first stream, topic, partition, or cons ### User login -Begin with the root account (`iggy` / `iggy`): +Use the credentials configured when the server was initialized. These examples use the `iggy` / `iggy` setup from the introduction: ```csharp var response = await client.LoginUserAsync("iggy", "iggy"); @@ -225,7 +231,7 @@ var permissions = new Permissions ReadUsers = false, SendMessages = false }, - Streams = new Dictionary + Streams = new Dictionary { [0] = new StreamPermissions { @@ -235,7 +241,7 @@ var permissions = new Permissions ReadTopics = true, PollMessages = true, SendMessages = true, - Topics = new Dictionary + Topics = new Dictionary { [0] = new TopicPermissions { @@ -261,6 +267,7 @@ var users = await client.GetUsersAsync(); // Update name and/or status (both optional) await client.UpdateUserAsync(userId, userName: "renamed_user", status: UserStatus.Inactive); +userId = Identifier.String("renamed_user"); // Replace permissions await client.UpdatePermissionsAsync(userId, permissions); @@ -317,7 +324,7 @@ await client.CreateTopicAsync( partitionsCount: 3, compressionAlgorithm: CompressionAlgorithm.None, messageExpiry: TimeSpan.Zero, // null or TimeSpan.Zero = server default; TimeSpan.MaxValue = never expire - maxTopicSize: 0 // 0 = unlimited + maxTopicSize: 1024 * 1024 * 1024 // 1 GiB ); ``` @@ -335,17 +342,18 @@ await client.CreateTopicAsync( options: new TopicOptions { SegmentSize = 128 * 1024 * 1024, // segment size in bytes (multiple of 512) - EnforceFsync = true, // fsync every write to this topic's partitions - MessagesRequiredToSave = 1000, // flush the journal after this many messages + Durability = Durability.Persisted, + ConsumerOffsetDurability = Durability.Persisted, + MessagesRequiredToSave = 1000, // attempt a flush after this many messages SizeOfMessagesRequiredToSave = 32 * 1024 * 1024, // or after this many bytes, whichever trips first - PreallocateSegments = true // reserve SegmentSize bytes up front + PreallocateSegments = true // request filesystem preallocation }.ToDictionary() ); ``` -A property left `null` emits no key and keeps the server default. These keys are **settable at creation only**: `UpdateTopicAsync` refuses them by name, the same way it refuses a key outside the server's catalog, so a mistyped key fails the call instead of being silently ignored. Use [`DescribeOptionsAsync`](#option-catalog) to enumerate the keys a server accepts. +Nullable properties left `null` emit no key and keep the server default. `Durability` and `ConsumerOffsetDurability` are non-nullable, default independently to `Replicated`, and are always emitted by `ToDictionary()`. These keys are **settable at creation only**: `UpdateTopicAsync` refuses them by name, the same way it refuses a key outside the server's catalog, so a mistyped key fails the call instead of being silently ignored. Use [`DescribeOptionsAsync`](#option-catalog) to enumerate the keys a server accepts. -There is **no on-demand flush command**. Durability is configured per topic through `EnforceFsync` and the flush thresholds above. +There is **no on-demand flush command**. `Durability` controls message completion and `ConsumerOffsetDurability` controls explicit offset-store completion. `Persisted` waits for persistence on the required quorum; flush thresholds alone do not provide that guarantee. See [Durability](/docs/server/durability) and [Topic options](/docs/server/topic-options). ```csharp var topicId = Identifier.String("my-topic"); @@ -355,13 +363,16 @@ var topic = await client.GetTopicByIdAsync(streamId, topicId); var topics = await client.GetTopicsAsync(streamId); // Update (name required; compression, expiry, size optional) -await client.UpdateTopicAsync(streamId, topicId, "renamed-topic"); +await client.UpdateTopicAsync(streamId, topicId, "renamed-topic", + messageExpiry: TimeSpan.FromDays(7), maxTopicSize: 1024 * 1024 * 1024); // Purge (delete all messages, keep the topic) / delete await client.PurgeTopicAsync(streamId, Identifier.String("renamed-topic")); await client.DeleteTopicAsync(streamId, Identifier.String("renamed-topic")); ``` +TCP omits zero expiry/size defaults from a topic update; HTTP sends them. Use explicit values when updating retention and review the [default-sentinel caveat](/docs/server/topic-options). + ## Partitions Add partitions to or remove them from an existing topic: @@ -402,7 +413,7 @@ foreach (var confirmation in response.Confirmations) } ``` -`SendMessagesAsync` returns a `SendMessagesResponse` whose `Confirmations` carry `StreamId`, `TopicId`, `PartitionId`, and `BaseOffset` (the offset of the first message of the batch) per committed batch. An empty list means the batch committed with no offsets to report. A single-message overload is also available: `SendMessagesAsync(streamId, topicId, partitioning, message)`. +`SendMessagesAsync` returns a `SendMessagesResponse` whose `Confirmations` carry `StreamId`, `TopicId`, `PartitionId`, and `BaseOffset` (the offset of the first message of the batch) per committed batch. An empty list reports no assigned offsets. A single-message overload is also available: `SendMessagesAsync(streamId, topicId, partitioning, message)`. To send many payloads without a `byte[]` allocation per message, build them into a single pooled buffer with `RentedMessageBatchBuilder` - see [Publishing with rented batches](/docs/sdk/csharp/high-level-sdk#publishing-with-rented-batches). @@ -412,20 +423,20 @@ Control which partition receives each message: ```csharp // Balanced - the client round-robins across partitions (default) -Partitioning.None() +Partitioning.None(); // Send to a specific partition (partition ids are 0-based) -Partitioning.PartitionId(0) +Partitioning.PartitionId(0); // Key-based routing - messages with the same key land on the same partition -Partitioning.EntityIdString("user-123") -Partitioning.EntityIdInt(12345) -Partitioning.EntityIdUlong(12345) -Partitioning.EntityIdGuid(Guid.NewGuid()) -Partitioning.EntityIdBytes(new byte[] { 1, 2, 3 }) +Partitioning.EntityIdString("user-123"); +Partitioning.EntityIdInt(12345); +Partitioning.EntityIdUlong(12345); +Partitioning.EntityIdGuid(Guid.NewGuid()); +Partitioning.EntityIdBytes(new byte[] { 1, 2, 3 }); ``` -Partition selection is resolved **client-side** on both transports. Balanced partitioning round-robins over the topic's partitions, and key-based routing hashes the key with xxHash32 modulo the partition count, consistent with the Rust SDK. +Partition selection is resolved **client-side** on both transports. Balanced partitioning round-robins over the topic's partitions, and key-based routing hashes the key with xxHash32 modulo the partition count, consistent with the Rust SDK for the same encoded key bytes and partition count. Strings use UTF-8, integers use little-endian bytes, and GUID keys use `Guid.ToByteArray()`. ### User-defined headers @@ -456,7 +467,7 @@ Available value factories: `FromString`, `FromBool`, `FromBytes`, `FromUInt8`, ` ### Fetching messages -Poll a batch of messages. The `partitionId` may be `null` to consume from any partition: +Poll a batch of messages. A null `partitionId` selects partition 0 for an ordinary consumer. For a group consumer, null lets the TCP client rotate through its assigned partitions: ```csharp var polledMessages = await client.PollMessagesAsync( @@ -490,9 +501,11 @@ var polledMessages = await client.PollMessagesAsync(new MessageFetchRequest }); ``` +With `autoCommit: true`, the server advances/submits the offset before application processing; the poll reply does not wait for durable completion of that offset store. Use `autoCommit: false` and explicitly store offsets after processing when that distinction matters. + ### Polling with rented buffers -`PollMessagesAsync` copies each payload into its own `byte[]`. On hot paths that allocation adds up. `PollMessagesRentedAsync` instead returns a `PolledMessagesRental` whose payloads and raw headers are slices over a single buffer rented from a shared pool - no per-message allocation. +On TCP, `PollMessagesAsync` copies each payload into its own `byte[]`. `PollMessagesRentedAsync` instead returns a `PolledMessagesRental` whose payloads and raw headers share a pooled buffer. Message wrappers and lazily parsed headers still allocate. HTTP first materializes its JSON response and then adapts it to the rental API. The rental owns that buffer, so you **must** dispose it, and the payload/header memory is only valid until you do. Wrap it in `using` and never hold a `Payload`/`RawUserHeaders` reference past the block: @@ -523,7 +536,7 @@ A `MessageFetchRequest` overload (`PollMessagesRentedAsync(request)`) is also av | Member | Type | Description | |--------|------|-------------| -| `PartitionId` | `int` | Partition the messages came from | +| `PartitionId` | `uint` | Partition the messages came from | | `CurrentOffset` | `ulong` | Current offset for the partition | | `Messages` | `IReadOnlyList` | The rented messages | @@ -545,11 +558,11 @@ The high-level `IggyConsumer` exposes the same pooled path as an async stream vi Control where consumption starts: ```csharp -PollingStrategy.Offset(1000) // from a specific offset -PollingStrategy.Timestamp(1699564800000000) // from a timestamp (microseconds since epoch) -PollingStrategy.First() // from the earliest message -PollingStrategy.Last() // from the latest message -PollingStrategy.Next() // from the next unread message +PollingStrategy.Offset(1000); // from a specific offset +PollingStrategy.Timestamp(1699564800000000); // from a timestamp (microseconds since epoch) +PollingStrategy.First(); // from the earliest message +PollingStrategy.Last(); // from the latest message +PollingStrategy.Next(); // from the next unread message ``` ## Offset management @@ -641,7 +654,8 @@ foreach (var spec in specs) `SendBinaryRequestAsync` is an escape hatch that sends a raw command code with a prebuilt payload and returns the raw response bytes, for commands the typed API does not cover: ```csharp -var responseBytes = await client.SendBinaryRequestAsync(commandCode, payloadBytes); +const uint PingCommand = 1; +var responseBytes = await client.SendBinaryRequestAsync(PingCommand, Array.Empty()); ``` Session-control codes (login, logout, and the like) are rejected with an invalid-command error, and the operation is TCP-only: HTTP clients throw `FeatureUnavailableException`. @@ -675,7 +689,7 @@ A full snapshot can exceed the default 64 MiB reply-frame bound, in which case t ### Segment management -Delete the last N segments from a partition: +Delete up to N of the oldest sealed segments from a partition. The active segment is retained: > **Note:** TCP-only - throws `FeatureUnavailableException` on HTTP. diff --git a/content/docs/sdk/csharp/high-level-sdk.mdx b/content/docs/sdk/csharp/high-level-sdk.mdx index ad8fd875b5..943285563d 100644 --- a/content/docs/sdk/csharp/high-level-sdk.mdx +++ b/content/docs/sdk/csharp/high-level-sdk.mdx @@ -10,21 +10,25 @@ The per-call [`IIggyClient`](/docs/sdk/csharp/guide) API is explicit but verbose - **Automatic offset commits** (on each poll, after each received message, or manual) - **Consuming a topic as an async stream** (`IAsyncEnumerable`) - **Typed message (de)serialization** -- Sending and receiving over **rented, pooled buffers** - no per-message allocation +- Sending and receiving over **rented, pooled buffers** to avoid separate payload copies on TCP -Both are built from an existing connected client via `IggyPublisherBuilder.Create(...)` / `IggyConsumerBuilder.Create(...)`, or equivalently via the `client.CreatePublisherBuilder(...)` / `client.CreateConsumerBuilder(...)` extension methods from `Apache.Iggy.Extensions`. +The fragments below reuse an authenticated `client` and the imports in the first example. The named streams/topics must already exist unless the snippet creates them. Both abstractions can be built from an existing connected client via `IggyPublisherBuilder.Create(...)` / `IggyConsumerBuilder.Create(...)`, or equivalently via the `client.CreatePublisherBuilder(...)` / `client.CreateConsumerBuilder(...)` extension methods from `Apache.Iggy.Extensions`. ## IggyPublisher Configure a publisher, initialize it, then send. `InitAsync` validates (and optionally creates) the stream and topic: ```csharp +using System.Buffers; +using System.Text; +using System.Text.Json; using Apache.Iggy; +using Apache.Iggy.Consumers; using Apache.Iggy.Kinds; using Apache.Iggy.Messages; using Apache.Iggy.Publishers; -var publisher = IggyPublisherBuilder.Create( +await using var publisher = IggyPublisherBuilder.Create( client, Identifier.String("my-stream"), Identifier.String("my-topic") @@ -46,19 +50,18 @@ await publisher.SendMessagesAsync(messages); // Drain the background queue, then dispose await publisher.WaitUntilAllSendsAsync(); -await publisher.DisposeAsync(); ``` ### Publisher builder options | Method | Description | |--------|-------------| -| `WithConnection(protocol, address, login, password, receiveBufferSize = 4096, sendBufferSize = 4096, reconnectionSettings = null)` | Connection settings - only used when the builder creates its own client (i.e. the `Create(streamId, topicId)` overload, without an existing client). The builder-created client auto-logs in with these credentials and replays them across reconnects | -| `WithConnection(protocol, address, personalAccessToken, receiveBufferSize = 4096, sendBufferSize = 4096, reconnectionSettings = null)` | Same, authenticating with a personal access token instead of a username and password | +| `WithConnection(protocol, address, login, password, receiveBufferSize = null, sendBufferSize = null, reconnectionSettings = null)` | Connection settings - only used when the builder creates its own client (i.e. the `Create(streamId, topicId)` overload, without an existing client). A builder-created TCP client auto-logs in and reuses the credentials across reconnects. For HTTP, supply an explicitly authenticated client | +| `WithConnection(protocol, address, personalAccessToken, receiveBufferSize = null, sendBufferSize = null, reconnectionSettings = null)` | Same, authenticating with a personal access token instead of a username and password | | `WithPartitioning(partitioning)` | Routing strategy for produced messages (default: balanced) | | `CreateStreamIfNotExists(name)` | Auto-create the stream on `InitAsync` if missing | | `CreateTopicIfNotExists(name, topicPartitionsCount = 1, compressionAlgorithm = None, messageExpiry = TimeSpan.Zero, maxTopicSize = 0)` | Auto-create the topic on `InitAsync` if missing | -| `WithRetry(enabled = true, maxAttempts = 3, initialDelay = 100ms, maxDelay = 10s, backoffMultiplier = 2.0)` | Retry failed sends with exponential backoff | +| `WithRetry(enabled = true, maxAttempts = 3, initialDelay = 100ms, maxDelay = 10s, backoffMultiplier = 2.0)` | Retry background sends with exponential backoff; direct sends surface errors | | `WithBackgroundSending(enabled = true, queueCapacity = 10000, batchSize = 100, flushInterval = 100ms, disposalTimeout = 5s)` | Queue and flush sends in the background for higher throughput | | `WithEncryptor(encryptor)` | Client-side payload encryption. Only valid on a builder-created client - for an external client, set `IggyClientConfigurator.MessageEncryptor` instead | | `SubscribeOnBackgroundError(handler)` | Observe background-processing errors (only fires when background sending is enabled) | @@ -70,7 +73,7 @@ await publisher.DisposeAsync(); | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | `enabled` | `bool` | `true` | Whether retry is enabled | -| `maxAttempts` | `int` | `3` | Maximum retry attempts | +| `maxAttempts` | `int` | `3` | Total send attempts, including the first send | | `initialDelay` | `TimeSpan?` | `100ms` | Delay before the first retry | | `maxDelay` | `TimeSpan?` | `10s` | Maximum delay between retries | | `backoffMultiplier` | `double` | `2.0` | Exponential backoff multiplier | @@ -81,15 +84,15 @@ await publisher.DisposeAsync(); |-----------|------|---------|-------------| | `enabled` | `bool` | `true` | Whether background sending is enabled | | `queueCapacity` | `int` | `10000` | Max queued send calls (one slot per `SendMessagesAsync` call, regardless of batch size) | -| `batchSize` | `int` | `100` | Messages sent per batch | +| `batchSize` | `int` | `100` | Flush threshold in messages; an individual send call is not split | | `flushInterval` | `TimeSpan?` | `100ms` | Interval at which pending messages are flushed | -| `disposalTimeout` | `TimeSpan?` | `5s` | How long `DisposeAsync` waits for the background processor to drain | +| `disposalTimeout` | `TimeSpan?` | `5s` | How long `DisposeAsync` waits for the processor to stop; pending sends may be discarded | -A batch also flushes once its accumulated payload reaches `IggyPublisherConfig.BackgroundMaxBatchBytes` (default 256 KB, `0` disables the byte gate) - whichever of `batchSize` or the byte limit is hit first. This knob has no builder method. Set it on the config directly if needed. +A batch also flushes once its accumulated payload reaches `IggyPublisherConfig.BackgroundMaxBatchBytes` (default 256 KiB, `0` disables the byte gate) - whichever of `batchSize` or the byte limit is hit first. This knob has no builder method. Set it on the config directly if needed. Await `WaitUntilAllSendsAsync` before disposal to finish queued sends; disposal alone can discard them. Background send calls return an empty confirmation list before delivery, and failures are reported through the subscribed events. ### Publishing with rented batches -`SendMessagesAsync` takes messages whose payloads each live in their own `byte[]`. On hot paths, build a `RentedMessageBatch` instead - every payload is written into a single buffer rented from the shared array pool - and hand it to `SendAsync`: +`SendMessagesAsync` accepts messages with `ReadOnlyMemory` payloads, which can share backing storage. On hot paths, build a `RentedMessageBatch` instead - every payload is written into a single buffer rented from the shared array pool - and hand it to `SendAsync`: ```csharp using System.Text.Json; @@ -109,6 +112,7 @@ foreach (var evt in events) var batch = builder.Build(); await publisher.SendAsync(batch); +await publisher.WaitUntilAllSendsAsync(); ``` `RentedMessageBatchBuilder`: @@ -135,13 +139,7 @@ Ownership rules: For automatic object serialization, use `IggyPublisherBuilder` with an `ISerializer`: ```csharp -class OrderSerializer : ISerializer -{ - public void Serialize(Order data, IBufferWriter writer) => - writer.Write(JsonSerializer.SerializeToUtf8Bytes(data)); -} - -var publisher = IggyPublisherBuilder.Create( +await using var publisher = IggyPublisherBuilder.Create( client, Identifier.String("orders-stream"), Identifier.String("orders-topic"), @@ -149,7 +147,15 @@ var publisher = IggyPublisherBuilder.Create( ).Build(); await publisher.InitAsync(); -await publisher.SendAsync(new List { /* ... */ }); +await publisher.SendAsync(new List { new(Guid.NewGuid(), 99.90m) }); + +record Order(Guid OrderId, decimal Amount); + +class OrderSerializer : ISerializer +{ + public void Serialize(Order data, IBufferWriter writer) => + writer.Write(JsonSerializer.SerializeToUtf8Bytes(data)); +} ``` Besides the collection overload, `SendAsync` also accepts a single item (`SendAsync(order, messageId: null, userHeaders: null)`) or a collection of `(data, messageId, userHeaders)` tuples when you need per-message ids or headers. @@ -157,7 +163,7 @@ Besides the collection overload, `SendAsync` also accepts a single item (`SendAs For JSON you don't need a custom serializer - the built-in `SystemTextJsonSerializer` (in `Apache.Iggy.Publishers`) writes System.Text.Json output directly into the send buffer, with optional `JsonSerializerOptions`: ```csharp -var publisher = IggyPublisherBuilder.Create( +await using var publisher = IggyPublisherBuilder.Create( client, Identifier.String("orders-stream"), Identifier.String("orders-topic"), @@ -165,6 +171,8 @@ var publisher = IggyPublisherBuilder.Create( ).Build(); ``` +With background sending enabled, typed values are serialized at flush time. Keep them immutable or pass a snapshot until `WaitUntilAllSendsAsync` completes. + There is **no built-in deserializer counterpart** - consumers implement `IDeserializer` themselves (see [Typed consumer](#typed-consumer)). ## IggyConsumer @@ -177,7 +185,7 @@ using Apache.Iggy; using Apache.Iggy.Consumers; using Apache.Iggy.Kinds; -var consumer = IggyConsumerBuilder.Create( +await using var consumer = IggyConsumerBuilder.Create( client, Identifier.String("my-stream"), Identifier.String("my-topic"), @@ -201,9 +209,9 @@ await foreach (var message in consumer.ReceiveAsync()) | Method | Description | |--------|-------------| -| `WithConnection(protocol, address, login, password, receiveBufferSize = 4096, sendBufferSize = 4096, reconnectionSettings = null)` | Connection settings - only used when the builder creates its own client. The builder-created client auto-logs in with these credentials and replays them across reconnects | -| `WithConnection(protocol, address, personalAccessToken, receiveBufferSize = 4096, sendBufferSize = 4096, reconnectionSettings = null)` | Same, authenticating with a personal access token instead of a username and password | -| `WithPartitionId(partitionId)` | Consume from a specific partition | +| `WithConnection(protocol, address, login, password, receiveBufferSize = null, sendBufferSize = null, reconnectionSettings = null)` | Connection settings - only used when the builder creates its own client. A builder-created TCP client auto-logs in and reuses the credentials across reconnects. For HTTP, supply an explicitly authenticated client | +| `WithConnection(protocol, address, personalAccessToken, receiveBufferSize = null, sendBufferSize = null, reconnectionSettings = null)` | Same, authenticating with a personal access token instead of a username and password | +| `WithPartitionId(partitionId)` | Ordinary consumer partition (null selects 0); ignored for consumer groups | | `WithPollingStrategy(pollingStrategy)` | Where to start consuming (default: `Offset(0)`). An `Offset(...)` strategy is advanced client-side after each poll; other strategies (e.g. `Next()`) are sent as-is and rely on server-side offset tracking | | `WithBatchSize(batchSize)` | Messages fetched per poll (default: `100`) | | `WithAutoCommitMode(mode)` | Offset auto-commit behavior (see below) | @@ -213,12 +221,14 @@ await foreach (var message in consumer.ReceiveAsync()) | `SubscribeOnPollingError(handler)` | Observe polling errors | | `WithLogger(loggerFactory)` | Logger factory for diagnostics | +For consumer groups, use `Next()` with offset commits to track each partition independently. An `Offset(...)` strategy advances one shared cursor across the group's partitions and can skip messages from another partition. + Auto-commit modes (`AutoCommitMode`): | Mode | Description | |------|-------------| -| `Auto` | Commit the offset while polling | -| `AfterReceive` | Commit after each message is received | +| `Auto` | Advance/submit the offset during the server poll, before application processing | +| `AfterReceive` | Store after the yielded message has been handled and enumeration resumes; breaking or throwing can leave that message uncommitted | | `Disabled` | Commit manually | ### Manual offset control @@ -233,14 +243,14 @@ await foreach (var message in consumer.ReceiveAsync()) } ``` -`StoreOffsetAsync(offset, partitionId, resetLastPolled = false)` stores the offset for a partition; pass `resetLastPolled: true` to also move the consumer's cached last-polled position so the next poll resumes past the stored offset. `DeleteOffsetAsync(partitionId)` clears the stored offset. +`StoreOffsetAsync(offset, partitionId, resetLastPolled = false)` stores the offset for a partition. `resetLastPolled: true` also changes its duplicate-filter position; it does not change the configured polling strategy or remove buffered messages. `DeleteOffsetAsync(partitionId)` clears the server offset while preserving that local state. Create a new consumer to restart with a different position. ### Consumer groups Pass a `Consumer.Group(...)` and let the builder create and join the group for load-balanced consumption: ```csharp -var consumer = IggyConsumerBuilder.Create( +await using var consumer = IggyConsumerBuilder.Create( client, Identifier.String("my-stream"), Identifier.String("my-topic"), @@ -259,12 +269,11 @@ await foreach (var message in consumer.ReceiveAsync()) Console.WriteLine($"Partition {message.PartitionId}: {payload}"); } -await consumer.DisposeAsync(); ``` ### Consuming with rented buffers -`ReceiveAsync` copies each payload into its own `byte[]`. `ReceiveRentedAsync` is the high-level counterpart of the low-level [rented poll API](/docs/sdk/csharp/guide#polling-with-rented-buffers): payloads are slices of a pooled buffer shared by all messages from the same poll. Each yielded `ReceivedRentedMessage` **must be disposed** - the buffer returns to the pool once the last message of its batch is disposed: +On TCP, `ReceiveAsync` copies each payload into its own `byte[]`. `ReceiveRentedAsync` is the high-level counterpart of the low-level [rented poll API](/docs/sdk/csharp/guide#polling-with-rented-buffers): payloads are slices of a pooled buffer shared by all messages from the same poll. Each yielded `ReceivedRentedMessage` **must be disposed** - the buffer returns to the pool once the last message of its batch is disposed: ```csharp await foreach (var message in consumer.ReceiveRentedAsync()) @@ -279,19 +288,13 @@ await foreach (var message in consumer.ReceiveRentedAsync()) `ReceivedRentedMessage` (`IDisposable`) exposes `Message` (a [`RentedMessageResponse`](/docs/sdk/csharp/guide#polling-with-rented-buffers)), `CurrentOffset`, `PartitionId`, `Status`, and `Error`. This path performs no deserialization, so `Status` is always `Success`. Payload and raw-header memory are only valid until the message is disposed - copy out anything you need to keep (e.g. `message.Message.Payload.ToArray()`). -Auto-commit modes apply exactly as with `ReceiveAsync`. Forgetting to dispose a message *never corrupts data* - the batch buffer simply isn't returned to the pool and is reclaimed by the GC as an ordinary allocation. +Auto-commit modes apply exactly as with `ReceiveAsync`. If a message is not disposed, the batch buffer is not returned to the pool; it becomes eligible for garbage collection once no references remain. ### Typed consumer For automatic deserialization, use `IggyConsumerBuilder` with an `IDeserializer` and iterate with `ReceiveDeserializedAsync`. Each `ReceivedMessage` carries a `Status` you should check: ```csharp -class OrderDeserializer : IDeserializer -{ - public OrderEvent Deserialize(ReadOnlyMemory data) => - JsonSerializer.Deserialize(data.Span)!; -} - var builder = IggyConsumerBuilder.Create( client, Identifier.String("orders-stream"), @@ -299,9 +302,10 @@ var builder = IggyConsumerBuilder.Create( Consumer.Group("order-processors"), new OrderDeserializer() ); +builder.WithPollingStrategy(PollingStrategy.Next()); builder.WithAutoCommitMode(AutoCommitMode.AfterReceive); -var consumer = builder.Build(); +await using var consumer = builder.Build(); await consumer.InitAsync(); await foreach (var message in consumer.ReceiveDeserializedAsync()) @@ -311,6 +315,14 @@ await foreach (var message in consumer.ReceiveDeserializedAsync()) Console.WriteLine($"Order: {message.Data?.OrderId}"); } } + +record OrderEvent(Guid OrderId, decimal Amount); + +class OrderDeserializer : IDeserializer +{ + public OrderEvent Deserialize(ReadOnlyMemory data) => + JsonSerializer.Deserialize(data.Span)!; +} ``` > **Note:** On the typed builders, the fluent `With*` methods are inherited from the untyped base builder and return the base type, while the typed `Build()` hides the base one. Chaining `Create(...).WithAutoCommitMode(...).Build()` therefore resolves to the base `Build()` and returns an **untyped** consumer/publisher. Keep the typed builder in a variable (as above) and call `Build()` on it - the `With*` calls mutate the builder, so their return value can be ignored. diff --git a/content/docs/sdk/csharp/intro.mdx b/content/docs/sdk/csharp/intro.mdx index 7e8ae6e302..916b05c41c 100644 --- a/content/docs/sdk/csharp/intro.mdx +++ b/content/docs/sdk/csharp/intro.mdx @@ -10,10 +10,23 @@ The SDK is built around the `IIggyClient` interface, which aggregates every feat ## Installation ```bash -dotnet add package Apache.Iggy --prerelease +dotnet add package Apache.Iggy --version 0.9.0-edge.9 ``` -The SDK targets .NET 8 and .NET 10 (`net8.0` and `net10.0`). Stable versions on NuGet stop at `0.8.0`, which predates the current wire protocol and cannot talk to a current server. The `0.9.0` line ships as **prerelease versions** (`0.9.0-edge.x` at the time of writing), so `--prerelease` is required. +The SDK targets .NET 8 and .NET 10 (`net8.0` and `net10.0`). Stable `0.8.0` uses TCP framing that predates the server 0.9.0 protocol. These pages prepare for server 0.9.0 and use the `0.9.0-edge.9` prerelease, which includes both topic durability options. When building from source, use the server and SDK from the same checkout. + +To use the source SDK with these release-preparation examples, run from the Iggy checkout root: + +```bash +dotnet new console --name IggySample --framework net10.0 +dotnet add IggySample/IggySample.csproj reference foreign/csharp/Iggy_SDK/Iggy_SDK.csproj +``` + +Put a sample in `IggySample/Program.cs` and run `dotnet run --project IggySample`. For console logging, add its package to that project: + +```bash +dotnet add IggySample/IggySample.csproj package Microsoft.Extensions.Logging.Console --version 10.0.11 +``` ## Supported protocols @@ -26,57 +39,62 @@ Some operations are **TCP-only** and throw `FeatureUnavailableException` on HTTP ## Connection semantics -Over TCP every request is wrapped in a 256-byte consensus header (Viewstamped Replication), the client registers a consensus session at login, and writes are replicated before they are acknowledged. The `IIggyClient` surface is unchanged. What changes is the behavior around connections, groups, and failures: +Over TCP every request is wrapped in a 256-byte consensus header (Viewstamped Replication), the client registers a consensus session at login, and replicated writes complete after the required quorum acknowledges them. Topic `Durability.Persisted` additionally waits for local persistence on that quorum; message and explicit offset durability are configured independently. See [Durability](/docs/server/durability). Both transports implement `IIggyClient`, with these TCP connection rules: - **Login binds a session.** `LoginUserAsync` / `LoginWithPersonalAccessTokenAsync` run the register handshake, and the session lives for as long as the connection. Logging out, being evicted, or losing the connection ends it, and the next login registers a fresh one. - **Leader redirection is automatic.** The client reads the cluster roster, follows the current leader, and re-checks it when a request is refused because the node stopped being primary. `GetCurrentAddress()` reports the node the client currently talks to. -- **The client picks partitions.** Balanced and message-key partitioning are resolved client-side (the message-key hash matches the Rust SDK byte for byte), and consumer-group polls round-robin over the partitions the coordinator assigned to this client. +- **The client picks partitions.** Balanced and message-key partitioning are resolved client-side (identical encoded key bytes and partition counts produce the same xxHash32 routing result as the Rust SDK), and consumer-group polls round-robin over the partitions the coordinator assigned to this client. - **Consumer groups are assignment-based.** `JoinConsumerGroupAsync` makes this client a member. The assignment is synced on demand and refreshed on every `PingAsync`. The TCP client pings on its own every `HeartbeatInterval` (5 seconds by default), so an idle session survives the server's heartbeat verification and assignments stay fresh. - **Credentials are bounds-checked locally.** A username outside 3-50 bytes, a password outside 3-100 bytes, or a personal access token outside 1-255 bytes is rejected before the request is framed. - **Consumer offsets need an explicit partition.** `StoreOffsetAsync` / `DeleteOffsetAsync` do not accept a `null` partition id under VSR. Passing one throws client-side. -- **Polling a topic that does not exist returns an empty poll** rather than throwing. The server answers an unresolved topic with the empty-poll reply shape, so the client cannot tell it apart from a topic with no messages. Check the topic exists first if the distinction matters. +- **Polling a missing topic throws `IggyInvalidStatusCodeException`.** An existing topic with no available messages returns an empty poll. ### Failed requests -The SDK replays a request whenever the server says it never admitted it. Two exceptions surface to the caller: +The SDK can replay a request the server reports as never admitted, within its request deadline. Replay-safe operations, including read-only polls, can also be retried after a lost connection. Two exceptions surface to the caller: - `IggyInvalidStatusCodeException` carries the server status code, with `FromServer` telling apart a verdict the cluster reported from a failure the client raised itself. -- `VsrRequestOutcomeUnknownException` means no server verdict arrived after the request was written (the connection was lost, the call was cancelled, or the server evicted the session mid-flight), so the cluster may or may not have committed it. The SDK will not replay it on a new session, because that would bypass server-side deduplication. Re-issuing it is **the caller's decision**. +- `VsrRequestOutcomeUnknownException` means no definitive server verdict arrived for an operation that cannot safely be replayed after it was written (the connection was lost, the call was cancelled, or the server evicted the session mid-flight), so the cluster may or may not have committed it. The SDK will not replay it on a new session, because re-issuing a possibly committed operation can repeat its effects. Re-issuing it is **the caller's decision**. The high-level abstractions handle the unknown-outcome case differently: -- `IggyConsumer` rethrows it rather than swallowing it, because an auto-committing poll may already have advanced the offset. Rethrowing ends the polling loop, so catch it around the enumeration, decide whether the operation is safe to re-issue, and start consuming again: +- `IggyConsumer` rethrows it rather than swallowing it, because an auto-committing poll may already have advanced the offset. Rethrowing ends the polling loop, so handle it around the enumeration. This fragment assumes an initialized `consumer` and a `CancellationToken token`, and stops consumption for an application-level recovery decision: ```csharp - while (!token.IsCancellationRequested) + using Apache.Iggy.Exceptions; + + try { - try - { - await foreach (var message in consumer.ReceiveAsync(token)) - { - Process(message); - } - } - catch (VsrRequestOutcomeUnknownException) + await foreach (var message in consumer.ReceiveAsync(token)) { - // The last poll may or may not have committed. Decide whether - // re-polling is safe for this workload, then resume the loop. + Console.WriteLine($"Offset {message.CurrentOffset}"); } } + catch (VsrRequestOutcomeUnknownException exception) + { + Console.Error.WriteLine($"Poll outcome is unknown: {exception.Message}"); + throw; + } ``` - `IggyPublisher` with background sending does not retry it: the batch is reported through the message-batch-failed event (`SubscribeOnMessageBatchFailed`), typed so a subscriber can tell "not sent" from "possibly sent twice". Direct (non-background) sends throw it to the caller. ## Creating a client -Create a client with `IggyClientFactory.CreateClient`, then call `ConnectAsync`: +The examples use `iggy` / `iggy`. For a new server data directory, start the matching source server from the Iggy repository root: + +```bash +IGGY_ROOT_USERNAME=iggy IGGY_ROOT_PASSWORD=iggy cargo run --bin iggy-server +``` + +For an existing server, use its configured credentials. Create a client with `IggyClientFactory.CreateClient`, then call `ConnectAsync`: ```csharp using Apache.Iggy.Configuration; using Apache.Iggy.Enums; using Apache.Iggy.Factory; -var client = IggyClientFactory.CreateClient(new IggyClientConfigurator +using var client = IggyClientFactory.CreateClient(new IggyClientConfigurator { BaseAddress = "127.0.0.1:8090", Protocol = Protocol.Tcp @@ -86,19 +104,22 @@ await client.ConnectAsync(); await client.LoginUserAsync("iggy", "iggy"); ``` -Optionally, provide an `ILoggerFactory` for diagnostics (defaults to `NullLoggerFactory.Instance`): +Optionally, provide an `ILoggerFactory` for diagnostics (defaults to `NullLoggerFactory.Instance`). The console logger requires the `Microsoft.Extensions.Logging.Console` package: ```csharp +using Apache.Iggy.Configuration; +using Apache.Iggy.Enums; +using Apache.Iggy.Factory; using Microsoft.Extensions.Logging; -var loggerFactory = LoggerFactory.Create(builder => +using var loggerFactory = LoggerFactory.Create(builder => { builder .AddFilter("Apache.Iggy", LogLevel.Information) .AddConsole(); }); -var client = IggyClientFactory.CreateClient(new IggyClientConfigurator +using var client = IggyClientFactory.CreateClient(new IggyClientConfigurator { BaseAddress = "127.0.0.1:8090", Protocol = Protocol.Tcp, @@ -106,7 +127,7 @@ var client = IggyClientFactory.CreateClient(new IggyClientConfigurator }); ``` -`IggyClientConfigurator` also exposes buffer sizes, TLS, the heartbeat interval, the maximum response frame size, automatic reconnection with exponential backoff (on by default), auto-login (so you can skip the explicit `LoginUserAsync` call), and client-side message encryption. See [Client configuration](/docs/sdk/csharp/guide#client-configuration) for the full reference. +`IggyClientConfigurator` also exposes TCP socket buffers, TLS, heartbeat, response-frame limits, reconnection with exponential backoff (on by default), auto-login, and message encryption. HTTP clients require explicit login; the TCP connection settings are not applied to HTTP. See [Client configuration](/docs/sdk/csharp/guide#client-configuration) for the full reference. ## Quick start @@ -123,7 +144,7 @@ using Apache.Iggy.Extensions; using Apache.Iggy.Factory; using Apache.Iggy.Messages; -var client = IggyClientFactory.CreateClient(new IggyClientConfigurator +using var client = IggyClientFactory.CreateClient(new IggyClientConfigurator { BaseAddress = "127.0.0.1:8090", Protocol = Protocol.Tcp @@ -132,7 +153,7 @@ var client = IggyClientFactory.CreateClient(new IggyClientConfigurator await client.ConnectAsync(); await client.LoginUserAsync("iggy", "iggy"); -var publisher = client.CreatePublisherBuilder( +await using var publisher = client.CreatePublisherBuilder( Identifier.String("sample-stream"), Identifier.String("sample-topic")) .CreateStreamIfNotExists("sample-stream") @@ -147,7 +168,6 @@ for (var i = 0; i < 10; i++) await publisher.SendMessagesAsync(new List { new(Guid.NewGuid(), payload) }); } -await publisher.DisposeAsync(); ``` ### Consumer @@ -162,7 +182,7 @@ using Apache.Iggy.Extensions; using Apache.Iggy.Factory; using Apache.Iggy.Kinds; -var client = IggyClientFactory.CreateClient(new IggyClientConfigurator +using var client = IggyClientFactory.CreateClient(new IggyClientConfigurator { BaseAddress = "127.0.0.1:8090", Protocol = Protocol.Tcp @@ -171,7 +191,7 @@ var client = IggyClientFactory.CreateClient(new IggyClientConfigurator await client.ConnectAsync(); await client.LoginUserAsync("iggy", "iggy"); -var consumer = client.CreateConsumerBuilder( +await using var consumer = client.CreateConsumerBuilder( Identifier.String("sample-stream"), Identifier.String("sample-topic"), Consumer.New(1)) @@ -193,5 +213,5 @@ await foreach (var message in consumer.ReceiveAsync()) ## Next steps - [Guide](/docs/sdk/csharp/guide) - client configuration reference and the full API surface: auth, streams, topics, partitions, publishing, consuming, offsets, consumer groups, system operations -- [High-level SDK](/docs/sdk/csharp/high-level-sdk) - `IggyPublisher` / `IggyConsumer` with background sending, retries, auto-commit, typed (de)serialization, and pooled (rented) buffers for allocation-free hot paths +- [High-level SDK](/docs/sdk/csharp/high-level-sdk) - `IggyPublisher` / `IggyConsumer` with background sending, retries, auto-commit, typed (de)serialization, and pooled (rented) buffers to avoid separate payload copies on TCP - [Examples](/docs/sdk/csharp/examples) - producer, consumer-group, and typed-message samples, plus links to runnable projects diff --git a/content/docs/sdk/go/examples.mdx b/content/docs/sdk/go/examples.mdx index b51e663e08..b44a1be91b 100644 --- a/content/docs/sdk/go/examples.mdx +++ b/content/docs/sdk/go/examples.mdx @@ -7,17 +7,17 @@ A runnable getting-started example lives in the [examples/go](https://github.com ## Starting the server -The Go SDK speaks the VSR (Viewstamped Replication) wire protocol, so the examples run against the VSR server. The examples log in as `iggy`/`iggy`, and root credentials are applied **only on the very first startup**, when no data directory exists yet. Start the server with the default root credentials: +The Go SDK speaks the VSR (Viewstamped Replication) wire protocol. Build the SDK and server from the same checkout for unreleased changes. The examples require Go 1.25 or newer and log in as `iggy`/`iggy`. From the repository root, start a disposable development server with no `IGGY_ROOT_USERNAME` or `IGGY_ROOT_PASSWORD` overrides: ```bash cargo run --bin iggy-server -- --fresh --with-default-root-credentials ``` -If an example fails to log in, the server data directory (`local_data` by default) was created with different credentials: delete it and start the server again with the defaults. This setup is intended only for development and testing. +`--fresh` **wipes this replica's local data directory** (`local_data` by default). Environment credentials take precedence over the flag, and bootstrap settings do not replace recovered credentials. A fresh cluster replica can recover credentials from peers. This setup is intended only for development and testing. ## Getting started -The [producer](https://github.com/apache/iggy/blob/master/examples/go/getting-started/producer/main.go) creates `sample-stream` and `sample-topic` (if missing) and sends 5 batches of 10 messages to partition `0`. The [consumer](https://github.com/apache/iggy/blob/master/examples/go/getting-started/consumer/main.go) polls them from offset `0` and exits after 5 batches. Stream, topic, and partition IDs are **0-based**. +The [producer](https://github.com/apache/iggy/blob/master/examples/go/getting-started/producer/main.go) creates `sample-stream` and `sample-topic` (if missing) and sends 5 batches of 10 messages to partition `0`. The [consumer](https://github.com/apache/iggy/blob/master/examples/go/getting-started/consumer/main.go) polls them from offset `0` and exits after 5 batches. Stream, topic, and partition IDs are **0-based**. These repository examples use numeric stream and topic IDs `0`, so create no other streams or topics before running the producer on the fresh server. Run the producer first. Run both from the `examples/go` directory: @@ -26,7 +26,7 @@ go run ./getting-started/producer/main.go go run ./getting-started/consumer/main.go ``` -Run multiple producers and consumers at once to see how messages are distributed across clients. +Each consumer reads partition `0` independently, starting from offset `0` or the earliest retained message. Running several consumers reads the same messages; these examples do not join a consumer group. Both binaries accept the same flags: @@ -37,16 +37,16 @@ Both binaries accept the same flags: ## TLS -Start the server with TLS enabled, using the development certificates from the repository: +From the repository root, start a disposable server with TLS enabled and the development certificates: ```bash IGGY_TCP_TLS_ENABLED=true \ IGGY_TCP_TLS_CERT_FILE=core/certs/iggy_cert.pem \ IGGY_TCP_TLS_KEY_FILE=core/certs/iggy_key.pem \ -cargo run --bin iggy-server +cargo run --bin iggy-server -- --fresh --with-default-root-credentials ``` -Then run the examples with the TLS flags (paths relative to `examples/go`): +The same data and credential prerequisites apply. These certificates are for development only. Run the producer before the consumer with the TLS flags (paths relative to `examples/go`): ```bash go run ./getting-started/producer/main.go --tcp-server-address localhost:8090 --tls --tls-ca-file ../../core/certs/iggy_ca_cert.pem diff --git a/content/docs/sdk/go/intro.mdx b/content/docs/sdk/go/intro.mdx index e930662194..7a120bdf8b 100644 --- a/content/docs/sdk/go/intro.mdx +++ b/content/docs/sdk/go/intro.mdx @@ -7,24 +7,36 @@ The Iggy Go SDK is a client library for interacting with the Iggy server from Go ## Server compatibility -The SDK and the server must speak the same wire protocol. Tagged releases up to `v0.8.0` predate VSR (and the current API) and pair only with servers of the same era. The module on `master` speaks VSR only and requires a current server. When working from source, build the SDK and the server from the same repository checkout. The code on this page tracks `master`. +The SDK and the server must speak the same wire protocol. Tagged releases up to `v0.8.0` predate VSR (and the current API) and pair only with servers of the same era. The current SDK speaks VSR only. This page prepares for server `0.9.0`; use a compatible SDK release or edge version. When working from source, build the SDK and the server from the same repository checkout. ## Installation +The current source requires Go 1.25 or newer. Run these commands from your application module: + ```bash go get github.com/apache/iggy/foreign/go ``` -`go get` resolves to the latest tagged release. Until a post-VSR release is tagged, track `master` to talk to a current server: +`go get` without a version does not automatically select prereleases. For prerelease testing, pin a compatible VSR edge version, for example: ```bash -go get github.com/apache/iggy/foreign/go@master +go get github.com/apache/iggy/foreign/go@v0.9.0-edge.6 ``` +For unreleased changes, use the local SDK replacement shown in the [repository examples](/docs/sdk/go/examples). A published edge tag need not contain every change in your checkout. + ## Quick start The snippets below follow the [getting-started example](https://github.com/apache/iggy/tree/master/examples/go/getting-started). Stream, topic, and partition IDs are **0-based**: the first partition of a topic is partition `0`. +Run the server from the repository root with disposable development data and no root credential overrides: + +```bash +cargo run --bin iggy-server -- --fresh --with-default-root-credentials +``` + +`--fresh` **wipes this replica's local data directory**. `IGGY_ROOT_USERNAME` and `IGGY_ROOT_PASSWORD` take precedence over the flag; bootstrap settings do not replace recovered credentials, including credentials recovered from cluster peers. The snippets use `iggy`/`iggy`. Run the producer before the consumer. + ### Producer ```go @@ -183,28 +195,30 @@ func main() { continue } - offset += uint64(len(polled.Messages)) for _, message := range polled.Messages { fmt.Printf("Offset: %d, Payload: %s\n", message.Header.Offset, string(message.Payload)) + offset = message.Header.Offset + 1 } } } ``` -Polled message payloads and user headers alias the reply buffer, so retaining one message pins the whole reply. Copy the bytes out when they outlive the poll. +The consumer reads partition `0` independently and keeps its offset only in memory. Restarting reads from offset `0` again, or the earliest retained message if older messages were removed. + +Uncompressed payloads and user headers alias the reply buffer, so retaining their slices pins the whole reply. Copy those bytes to retain only the message you need. S2 decompression allocates a separate payload buffer. ## TLS -Wrap the TCP options with `tcp.WithTLS`: +Replace the quick-start client constructor with the following TCP options. Start the TLS server as shown on the [examples page](/docs/sdk/go/examples#tls); the development CA path below is relative to `examples/go`: ```go cli, err := client.NewIggyClient( client.WithTcp( - tcp.WithServerAddress("iggy.example.com:8090"), + tcp.WithServerAddress("localhost:8090"), tcp.WithTLS( - tcp.WithTLSCAFile("/path/to/ca.pem"), - tcp.WithTLSDomain("iggy.example.com"), + tcp.WithTLSCAFile("../../core/certs/iggy_ca_cert.pem"), + tcp.WithTLSDomain("localhost"), ), ), ) @@ -214,7 +228,9 @@ cli, err := client.NewIggyClient( ## Delivery semantics -`SendMessages` returns the placements the server committed. Delivery is at-least-once. A send whose reply is lost to a dropped connection returns `ErrDisconnected` without a replay: a reconnect registers a fresh client identity, so the server could not deduplicate the replay against a batch that may have already committed. Retrying such a send is the caller's decision and may write the batch twice. Consumers that need exactly-once handling deduplicate on the message id. A confirmation reports an in-memory commit, not a flush to disk, and an empty confirmation list is a valid success. +`SendMessages` returns any placements the server reports. A send whose reply is lost to a dropped connection returns `ErrDisconnected` without a replay: a reconnect registers a fresh client identity, so the server could not deduplicate the replay against a batch that may have already committed. Retrying such a send is the caller's decision and may write the batch twice. Consumers must handle duplicates through idempotent processing or application-level deduplication. + +Crash durability follows the topic's `durability` policy: `replicated` confirms replication, while `persisted` also waits for the required replicas to persist the message data. See [Durability](/docs/server/durability). An empty confirmation list is a valid success but does not by itself prove that new messages were appended. ## Consumer groups diff --git a/content/docs/sdk/introduction.mdx b/content/docs/sdk/introduction.mdx index 3ed4a3879b..ee2a8a7164 100644 --- a/content/docs/sdk/introduction.mdx +++ b/content/docs/sdk/introduction.mdx @@ -14,7 +14,7 @@ Iggy provides official client SDKs in multiple languages. The **Rust SDK** is th | Go | [iggy-go](https://pkg.go.dev/github.com/apache/iggy/foreign/go) | pkg.go.dev | TCP | No | | C# | [Apache.Iggy](https://www.nuget.org/packages/Apache.Iggy/) | NuGet | TCP, HTTP | No | | C++ | [iggy-cpp](https://github.com/apache/iggy/tree/master/foreign/cpp) | GitHub (WIP) | TCP, QUIC, HTTP, WebSocket | Yes (Rust FFI) | -| PHP | [apache/iggy-php](https://github.com/apache/iggy/tree/master/foreign/php) | Composer (experimental) | TCP, QUIC, HTTP, WebSocket | Yes (Rust FFI) | +| PHP | [apache/iggy-php](https://github.com/apache/iggy/tree/master/foreign/php) | GitHub (source) | TCP, QUIC, HTTP, WebSocket | Yes (Rust FFI) | Python, C++, and PHP wrap the Rust SDK, so they inherit all transport protocols when using connection strings. Java and C# implement TCP and HTTP natively. Go and Node.js currently support TCP *only*. @@ -22,13 +22,13 @@ The Rust SDK is always the first to receive new features. ## Prerequisites -The samples across the SDK docs log in with the `iggy`/`iggy` root credentials. These **only exist** when the server is started with the default-credentials flag (or `IGGY_ROOT_USERNAME`/`IGGY_ROOT_PASSWORD` set) on its first boot. Otherwise root gets a generated password and the samples fail with `InvalidCredentials`: +The examples below use the `iggy`/`iggy` root credentials. For a new local development instance, with no `IGGY_ROOT_USERNAME` or `IGGY_ROOT_PASSWORD` overrides, use: ```bash cargo run --bin iggy-server -- --fresh --with-default-root-credentials ``` -`--fresh` **wipes the local data directory**, so an existing server with different stored credentials is re-initialized. This setup is for *development only*. +`--fresh` **wipes this replica's local data directory**. In a cluster, it can recover stored credentials from another replica. The environment takes precedence over the default-credentials flag, and bootstrap settings do not replace recovered credentials. This setup is for *development only*; use a disposable data directory. See [Connection Strings](/docs/sdk/connection-strings) for the authentication details. ## Connection string @@ -58,29 +58,34 @@ iggy://@host:port **Examples:** +The Rust snippets assume an async Tokio context. HTTP requires an explicit login after construction; connection-string credentials are applied automatically only by the binary transports. + ```rust +use iggy::prelude::*; + // Rust - TCP with default options -let client = IggyClient::from_connection_string("iggy://iggy:iggy@localhost:8090")?; +let client = IggyClient::from_connection_string("iggy://iggy:iggy@127.0.0.1:8090")?; // Rust - QUIC -let client = IggyClient::from_connection_string("iggy+quic://iggy:iggy@localhost:8080")?; +let client = IggyClient::from_connection_string("iggy+quic://iggy:iggy@127.0.0.1:8080")?; // Rust - TCP with options let client = IggyClient::from_connection_string( - "iggy://iggy:iggy@localhost:8090?tls=true&reconnection_retries=unlimited&heartbeat_interval=5s" + "iggy://iggy:iggy@127.0.0.1:8090?nodelay=true&reconnection_retries=unlimited&heartbeat_interval=5s" )?; ``` ```python -# Python -client = IggyClient.from_connection_string("iggy://iggy:iggy@localhost:8090") +from apache_iggy import IggyClient + +client = IggyClient.from_connection_string("iggy://iggy:iggy@127.0.0.1:8090") ``` The option keys differ per transport and unknown keys are rejected as **hard errors**. See [Connection Strings](/docs/sdk/connection-strings) for the full per-transport option tables, defaults, and default ports. ## Common operations -All SDKs provide the same core operations through the unified client interface: +The Rust SDK exposes these operations through traits on `IggyClient`. Availability and method names vary across foreign SDKs and transports: - **SystemClient** - ping, stats, snapshot - **StreamClient** - create, get, list, update, delete, purge streams @@ -96,19 +101,19 @@ All SDKs provide the same core operations through the unified client interface: ## Polling strategies -When polling messages, you can choose from several strategies: +When polling messages, you can choose from several strategies (Rust method names are shown): | Strategy | Description | |----------|-------------| | `offset(n)` | Start from a specific offset | -| `timestamp(t)` | Start after a specific timestamp | +| `timestamp(t)` | Start at or after the given broker append timestamp | | `first()` | Start from the earliest available message | -| `last()` | Start from the latest message | -| `next()` | Continue from the last committed offset | +| `last()` | Read the tail ending at the committed offset, up to the requested count | +| `next()` | Continue after the stored consumer offset, or start at zero if none is stored | ## Partitioning strategies -When sending messages, you can control partition routing: +When sending messages, you can control partition routing (Rust method names are shown): | Strategy | Description | |----------|-------------| @@ -116,4 +121,4 @@ When sending messages, you can control partition routing: | `balanced()` | Round-robin across partitions | | `messages_key(key)` | Hash-based routing by key | -Strategies resolve in the client on binary transports, and on the server for the HTTP API and the Node SDK. +The Rust SDK resolves these strategies before sending a binary request. The Rust HTTP client and Node SDK send the strategy to the server for resolution. diff --git a/content/docs/sdk/java/examples.mdx b/content/docs/sdk/java/examples.mdx index e56e6df773..441b8abc47 100644 --- a/content/docs/sdk/java/examples.mdx +++ b/content/docs/sdk/java/examples.mdx @@ -3,21 +3,21 @@ title: Examples description: "Runnable Java examples from the core repository, built with Gradle, and how to start a server for them." --- -Runnable examples live in the [examples/java](https://github.com/apache/iggy/tree/master/examples/java) directory of the core repository as a standalone Gradle project. The project builds against the in-repo SDK (via an `includeBuild` substitution), so the examples always match the SDK source in the same checkout. Java 17 is recommended. The included `gradlew` wrapper downloads the pinned Gradle version on first run. +Runnable examples live in the [examples/java](https://github.com/apache/iggy/tree/master/examples/java) directory of the core repository as a standalone Gradle project. The project builds against the in-repo SDK (via an `includeBuild` substitution), so the examples always match the SDK source in the same checkout. Java 17 or newer is required. The included `gradlew` wrapper downloads the pinned Gradle version on first run. ## Starting the server -The Java SDK speaks the VSR (Viewstamped Replication) wire protocol, so the examples run against the VSR server. The examples log in as `iggy`/`iggy`, and root credentials are applied **only on the very first startup**, when no data directory exists yet. Start the server with the default root credentials: +These examples prepare for server **0.9.0** and speak the VSR (Viewstamped Replication) wire protocol. Build the SDK and server from the same checkout for unreleased changes. The examples log in as `iggy`/`iggy`. From the repository root, start a disposable development server with no `IGGY_ROOT_USERNAME` or `IGGY_ROOT_PASSWORD` overrides: ```bash cargo run --bin iggy-server -- --fresh --with-default-root-credentials ``` -Alternatively, set `IGGY_ROOT_USERNAME=iggy` and `IGGY_ROOT_PASSWORD=iggy` before the first start. If an example fails with `InvalidCredentials`, the server data directory (`local_data` by default) was created with different credentials: delete it and start the server again with the defaults. This setup is intended only for development and testing. +`--fresh` **wipes this replica's local data directory** (`local_data` by default). Environment credentials take precedence over the flag, and bootstrap settings do not replace recovered credentials. A fresh cluster replica can recover credentials from peers. This setup is intended only for development and testing. ## Running the examples -Run each example from the `examples/java` directory with its Gradle task: +Run each example from the `examples/java` directory with its Gradle task. Run the producer before its consumer. The ordinary consumers keep their cursor in memory and start from offset `0` or the earliest retained message on each run; the multi-tenant consumer uses consumer groups and server-managed offsets. - **[gettingstarted](https://github.com/apache/iggy/tree/master/examples/java/src/main/java/org/apache/iggy/examples/gettingstarted)** - basic blocking producer and consumer, the best starting point. @@ -26,7 +26,7 @@ Run each example from the `examples/java` directory with its Gradle task: ./gradlew runGettingStartedConsumer ``` -- **[messageheaders](https://github.com/apache/iggy/tree/master/examples/java/src/main/java/org/apache/iggy/examples/messageheaders)** - message metadata via custom header keys and values, with header-based routing instead of payload-based typing. +- **[messageheaders](https://github.com/apache/iggy/tree/master/examples/java/src/main/java/org/apache/iggy/examples/messageheaders)** - message metadata via custom header keys and values, with a `message_type` header selecting the application handler. ```bash ./gradlew runMessageHeadersProducer @@ -47,13 +47,13 @@ Run each example from the `examples/java` directory with its Gradle task: ./gradlew runMultiTenantConsumer ``` -- **[sinkdataproducer](https://github.com/apache/iggy/tree/master/examples/java/src/main/java/org/apache/iggy/examples/sinkdataproducer)** - high-volume data generation (1000+ messages per batch) with realistic records, for testing and benchmarking. +- **[sinkdataproducer](https://github.com/apache/iggy/tree/master/examples/java/src/main/java/org/apache/iggy/examples/sinkdataproducer)** - high-volume data generation (100 batches of 1000 to 1099 messages) with realistic records, for testing and benchmarking. ```bash ./gradlew runSinkDataProducer ``` -- **[streambuilder](https://github.com/apache/iggy/tree/master/examples/java/src/main/java/org/apache/iggy/examples/streambuilder)** - a combined producer and consumer in a single class. +- **[streambuilder](https://github.com/apache/iggy/tree/master/examples/java/src/main/java/org/apache/iggy/examples/streambuilder)** - a combined producer and consumer in a single class. It deletes its `test_stream` stream after the run. ```bash ./gradlew runStreamBasic @@ -66,15 +66,17 @@ Run each example from the `examples/java` directory with its Gradle task: ./gradlew runAsyncConsumer ``` -- **[tcptls](https://github.com/apache/iggy/tree/master/examples/java/src/main/java/org/apache/iggy/examples/tcptls)** - TLS-encrypted TCP connections with CA certificate verification. Requires a TLS-enabled server: +- **[tcptls](https://github.com/apache/iggy/tree/master/examples/java/src/main/java/org/apache/iggy/examples/tcptls)** - TLS-encrypted TCP connections with CA certificate verification. Requires a TLS-enabled server. Run this server command from the repository root: ```bash IGGY_TCP_TLS_ENABLED=true \ IGGY_TCP_TLS_CERT_FILE=core/certs/iggy_cert.pem \ IGGY_TCP_TLS_KEY_FILE=core/certs/iggy_key.pem \ - cargo run --bin iggy-server + cargo run --bin iggy-server -- --fresh --with-default-root-credentials ``` + The same data and credential prerequisites apply. These certificates are for development only. Return to `examples/java` for the client commands; their CA path is relative to that directory. + ```bash ./gradlew runTcpTlsProducer ./gradlew runTcpTlsConsumer diff --git a/content/docs/sdk/java/intro.mdx b/content/docs/sdk/java/intro.mdx index 651b1c0c2a..884aae570e 100644 --- a/content/docs/sdk/java/intro.mdx +++ b/content/docs/sdk/java/intro.mdx @@ -7,24 +7,45 @@ The Iggy Java SDK is a client library for interacting with the Iggy server from ## Server compatibility -The TCP transport speaks the VSR (Viewstamped Replication) wire protocol. The SDK and the server must speak the same protocol: the latest published artifact (`0.6.0`) predates VSR and only pairs with servers of the same era, while the SDK on `master` requires a current server. When working from source, build the SDK and the server from the same repository checkout. The code on this page tracks `master`. +This documentation prepares for server **0.9.0**. The TCP transport speaks the VSR (Viewstamped Replication) wire protocol. The released Java artifact `0.8.0` uses the older protocol; use the `0.9.0-SNAPSHOT` build below for the pre-release SDK. When working from source, build the SDK and the server from the same repository checkout. ## Installation +Use Java 17 or newer. These examples use the ASF snapshot repository until the `0.9.0` artifact is published to Maven Central. + ### Maven +Add the repository and dependency under the `project` element in `pom.xml`: + ```xml - - org.apache.iggy - iggy - 0.6.0 - + + + apache-snapshots + https://repository.apache.org/content/repositories/snapshots/ + true + + + + + + org.apache.iggy + iggy + 0.9.0-SNAPSHOT + + ``` ### Gradle ```groovy -implementation 'org.apache.iggy:iggy:0.6.0' +repositories { + mavenCentral() + maven { url = uri('https://repository.apache.org/content/repositories/snapshots/') } +} + +dependencies { + implementation 'org.apache.iggy:iggy:0.9.0-SNAPSHOT' +} ``` Check [Maven Central](https://central.sonatype.com/artifact/org.apache.iggy/iggy) for the latest published version. Snapshot builds are available from the [ASF snapshot repository](https://repository.apache.org/content/repositories/snapshots/). @@ -33,6 +54,8 @@ Check [Maven Central](https://central.sonatype.com/artifact/org.apache.iggy/iggy The snippets below follow the [getting-started example](https://github.com/apache/iggy/tree/master/examples/java/src/main/java/org/apache/iggy/examples/gettingstarted). Stream, topic, and partition IDs are **0-based**: the first partition of a topic is partition `0`. +Start a development server using the [example setup](/docs/sdk/java/examples#starting-the-server), with `iggy`/`iggy` credentials, then run the producer before the consumer. Bootstrap credentials do not replace recovered credentials. + ### Producer ```java @@ -135,14 +158,16 @@ public class SampleConsumer { for (Message message : polledMessages.messages()) { String payload = new String(message.payload(), StandardCharsets.UTF_8); System.out.printf("Offset: %d, Payload: %s%n", message.header().offset(), payload); + offset = message.header().offset().add(BigInteger.ONE); } - offset = offset.add(BigInteger.valueOf(polledMessages.messages().size())); } } } } ``` +The consumer reads partition `0` independently and stops on an empty poll. Its cursor starts at offset `0` or the earliest retained message each time it runs; `autoCommit` is disabled, so this sample does not store its progress on the server. + ## Client types and configuration `org.apache.iggy.Iggy` is the unified entrypoint for building clients. `Iggy.tcpClientBuilder().blocking()` returns the same builder as `IggyTcpClient.builder()` used above. @@ -175,18 +200,21 @@ var httpClient = Iggy.httpClientBuilder() .buildAndLogin(); ``` -The TCP builder also accepts TLS and resilience options: +Close clients when finished: `client.close()`, `asyncClient.close().join()`, and `httpClient.close()`. The HTTP close method declares `IOException`. + +The TCP builder also accepts TLS and resilience options. Start the [TLS development server](/docs/sdk/java/examples#running-the-examples) and run this fragment with `examples/java` as the working directory: ```java import java.time.Duration; +import org.apache.iggy.Iggy; import org.apache.iggy.config.RetryPolicy; var client = Iggy.tcpClientBuilder() .blocking() - .host("iggy.example.com") + .host("localhost") .port(8090) .enableTls() - .tlsCertificate("/path/to/ca.pem") // optional custom CA + .tlsCertificate("../../core/certs/iggy_ca_cert.pem") .connectionTimeout(Duration.ofSeconds(10)) .requestTimeout(Duration.ofSeconds(30)) .retryPolicy(RetryPolicy.exponentialBackoff()) @@ -194,7 +222,7 @@ var client = Iggy.tcpClientBuilder() .buildAndLogin(); ``` -Beyond streams, topics, and messages, the client exposes consumer groups, partitions, users, and personal access tokens through `client.consumerGroups()`, `client.partitions()`, `client.users()`, and `client.personalAccessTokens()`. All SDK exceptions inherit from `IggyException`. +Beyond streams, topics, and messages, the client exposes consumer groups, partitions, users, and personal access tokens through `client.consumerGroups()`, `client.partitions()`, `client.users()`, and `client.personalAccessTokens()`. The SDK's custom exception types inherit from `IggyException`; failed futures can wrap them in `CompletionException` when joined. ## Examples diff --git a/content/docs/sdk/node/examples.mdx b/content/docs/sdk/node/examples.mdx index caae3e0c15..a338baac0f 100644 --- a/content/docs/sdk/node/examples.mdx +++ b/content/docs/sdk/node/examples.mdx @@ -1,47 +1,52 @@ --- title: Examples -description: "TypeScript examples for the Node.js SDK, exercised by CI so they match the current SDK." +description: "Build the local Node.js SDK and run its TypeScript examples." --- -Working examples are available in the [examples/node](https://github.com/apache/iggy/tree/master/examples/node) directory, written in TypeScript. CI exercises them, so they always match the current SDK: +Examples are available in the [examples/node](https://github.com/apache/iggy/tree/master/examples/node) directory, written in TypeScript. They use the SDK from the same checkout: - **getting-started** - basic producer and consumer - **basic** - producer and consumer with utilities - **message-envelope** - JSON message envelope pattern -- **message-headers** - custom message headers -- **multi-tenant** - multi-tenant streaming setup +- **message-headers** - message-type dispatch using a JSON wrapper in the payload +- **multi-tenant** - separate streams for tenants, using one client - **tcp-tls** - TLS-encrypted TCP connections - **stream-builder** - end-to-end walkthrough: create a stream and topic, produce, and consume - **sink-data-producer** - bulk data generation for sink connectors ## Running the examples -Start an Iggy server first: +These examples target server **0.9.0**. Start the server from the same source checkout; see [Getting started](/docs/introduction/getting-started) for prerequisites: ```bash -# Latest release -docker run --rm \ - --cap-add=SYS_NICE --security-opt seccomp=unconfined --ulimit memlock=-1:-1 \ - -p 8090:8090 \ - -e IGGY_TCP_ADDRESS=0.0.0.0:8090 \ - -e IGGY_NODE_ADVERTISED_ADDRESS=localhost \ - -e IGGY_ROOT_USERNAME=iggy -e IGGY_ROOT_PASSWORD=iggy \ - apache/iggy:latest - -# Or from the repository source cargo run --bin iggy-server -- --fresh --with-default-root-credentials ``` -The environment variables make the server reachable through the published port (it binds to `127.0.0.1` inside the container by default) and set the `iggy`/`iggy` root credentials the examples default to. +The default-root flag sets the `iggy`/`iggy` credentials used by the examples. `--fresh` deletes existing local server data. Explicit credential environment variables override the default-root flag; bootstrap settings do not replace recovered credentials. -Then, from `examples/node`, install the dependencies and run a producer/consumer pair: +From the repository root, build the local SDK before installing the examples. The stream-builder example creates a stream and topic, sends and consumes three messages, and deletes its resources: ```bash -npm ci -npm run test:getting-started:producer -npm run test:getting-started:consumer +npm --prefix foreign/node ci +npm --prefix foreign/node run build +npm --prefix examples/node ci +cd examples/node +DEBUG=iggy:examples npm run test:stream-builder ``` -Each example set has a matching `test:` script in `examples/node/package.json`. +Each example has a script in `examples/node/package.json`. Set `DEBUG=iggy:examples*` to see its progress. + +The separate producer and consumer scripts do not consistently share resources: the envelope and headers consumers create new empty topics, and the tenant producer deletes its streams before exiting. The getting-started and basic producers spread batches over five partitions, while their consumers poll partition 0. Use the self-contained stream-builder command above for a complete send-and-consume walkthrough. The sink-data producer deletes its selected topic and stream when it finishes, including resources that already existed. + +The TCP/TLS pair needs a TLS-enabled source server. Run this command from the repository root, then the two `test:tcp-tls:*` scripts from `examples/node`, where they load `../../core/certs/iggy_ca_cert.pem`: + +```bash +IGGY_TCP_TLS_ENABLED=true \ +IGGY_TCP_TLS_CERT_FILE=core/certs/iggy_cert.pem \ +IGGY_TCP_TLS_KEY_FILE=core/certs/iggy_key.pem \ +cargo run --bin iggy-server -- --fresh --with-default-root-credentials +``` + +These are repository development certificates. For the quick-start code itself, see the [Node.js SDK intro](/docs/sdk/node/intro). diff --git a/content/docs/sdk/node/intro.mdx b/content/docs/sdk/node/intro.mdx index 62573825af..717a947b81 100644 --- a/content/docs/sdk/node/intro.mdx +++ b/content/docs/sdk/node/intro.mdx @@ -5,33 +5,35 @@ description: "The Node.js and TypeScript SDK, from installation to keeping clien The Iggy Node.js SDK is a client library that allows you to interact with the Iggy API from your Node.js and TypeScript applications. It communicates with the Iggy server over TCP or TLS using the binary protocol. The package is available on [npm](https://www.npmjs.com/package/apache-iggy) and the source code can be found on [GitHub](https://github.com/apache/iggy/tree/master/foreign/node). -The SDK speaks only the current (VSR) wire protocol and doesn't fall back to older formats. Keep the client and server versions in step: pair the newest package with the newest server. +These docs target server **0.9.0**. Use the Node SDK's `edge` package with server 0.9.0 or `edge`; the stable Node package 0.8.0 uses the older protocol. The current SDK speaks only the VSR wire protocol and doesn't fall back to older formats. ## Installation ```bash -npm install apache-iggy +npm install apache-iggy@edge ``` ## Quick start -The samples below expect an Iggy server on `127.0.0.1:8090`: +The samples below expect an Iggy server on `127.0.0.1:8090`. See [Getting started](/docs/introduction/getting-started) for the server prerequisites: ```bash -# Latest release +# Development image for the 0.9.0 release docker run --rm \ --cap-add=SYS_NICE --security-opt seccomp=unconfined --ulimit memlock=-1:-1 \ -p 8090:8090 \ -e IGGY_TCP_ADDRESS=0.0.0.0:8090 \ -e IGGY_NODE_ADVERTISED_ADDRESS=localhost \ -e IGGY_ROOT_USERNAME=iggy -e IGGY_ROOT_PASSWORD=iggy \ - apache/iggy:latest + apache/iggy:edge -# Or from the repository source +# Or from the matching repository root cargo run --bin iggy-server -- --fresh --with-default-root-credentials ``` -The environment variables make the server reachable through the published port (it binds to `127.0.0.1` inside the container by default) and set the root credentials the samples log in with. Without `--with-default-root-credentials` (or the `IGGY_ROOT_USERNAME` / `IGGY_ROOT_PASSWORD` variables), a first boot generates a random root password instead of `iggy`/`iggy`. +The environment variables make the server reachable through the published port (it binds to `127.0.0.1` inside the container by default) and set the root credentials the samples log in with. Without `--with-default-root-credentials` (or the `IGGY_ROOT_USERNAME` / `IGGY_ROOT_PASSWORD` variables), a first boot generates a random root password instead of `iggy`/`iggy`. For source runs, `--fresh` deletes existing local server data. Explicit credential environment variables take precedence over the default-root flag; bootstrap settings do not replace credentials already stored in recovered data. + +Save the following snippets as `producer.mjs` and `consumer.mjs` in the project where you installed the SDK. Run `node producer.mjs` before `node consumer.mjs`. ### Producer @@ -81,7 +83,7 @@ console.log(`Sent ${messages.length} message(s)`); await client.destroy(); ``` -The Node SDK sends the partitioning choice on the wire, and the server resolves balanced and key routing at admission (the Rust, C#, Java, and Go clients instead resolve them client-side). +The Node SDK sends the partitioning choice on the wire, and the server resolves balanced and key routing at admission (the Rust, C#, Java, and Go binary clients instead resolve them client-side). ### Consumer @@ -98,8 +100,8 @@ const client = new Client({ credentials: { username: 'iggy', password: 'iggy' }, }); -// Next with autocommit continues from this consumer's last committed offset, -// so each run picks up where the previous one finished. +// Next starts after consumer 0's stored offset. +// Autocommit requests offset storage before application processing. const polledMessages = await client.message.poll({ streamId: STREAM_NAME, topicId: TOPIC_NAME, @@ -118,19 +120,21 @@ for (const message of polledMessages.messages) { await client.destroy(); ``` +The poll response does not wait for the auto-commit to finish. The sample processes messages after polling, so auto-commit does not confirm that the application handled each message. + ## Configuration The client constructor accepts a few options: - **reconnect**: automatic reconnection with `{ enabled, interval, maxRetries }`. -- **heartbeatInterval**: the client pings the server every 5 seconds by default. `0` disables heartbeats. The server evicts connections that stay silent too long, so keep it well below the server's heartbeat timeout. +- **heartbeatInterval**: the client pings the server every 5 seconds by default. `0` disables heartbeats. When server heartbeat eviction is enabled, silent consumer-group members can be evicted, so keep it well below the server's heartbeat timeout. - **maxResponseFrameSize**: response frames larger than this limit (default 64 MiB) are rejected and close the connection. Raise it when polling very large batches. - **poolSize**: the client currently supports exactly one pooled connection. `min`/`max` above 1 throw. - **TLS**: set `transport: 'TLS'` and pass Node `tls.ConnectionOptions` in `options`. See the [tcp-tls example](https://github.com/apache/iggy/tree/master/examples/node/src/tcp-tls). ## Beyond the basics -- **Consumer groups**: `client.group` manages groups (including ensure-and-join in one call), and polling with `Consumer.Group` distributes partitions across group members. +- **Consumer groups**: `client.group` manages groups (including ensure-and-join in one call), and polling with `Consumer.Group(groupId)` and `partitionId: null` follows the partitions assigned to that member. - **Consumer streams**: `singleConsumerStream` and `groupConsumerStream` expose polling as Node streams. - **Administration**: users, personal access tokens, offsets, partitions, segments, and cluster metadata are all available on the client. - **Errors**: the package exports `ResponseError`, `DeserializeError`, `ProtocolFrameError`, and `VsrEvictionError` for precise error handling. @@ -139,4 +143,4 @@ See the [foreign/node README](https://github.com/apache/iggy/tree/master/foreign ## Examples -Working examples are available in the [examples/node](https://github.com/apache/iggy/tree/master/examples/node) directory, written in TypeScript. See [Examples](/docs/sdk/node/examples) for the list and how to run them. +Examples are available in the [examples/node](https://github.com/apache/iggy/tree/master/examples/node) directory, written in TypeScript. See [Examples](/docs/sdk/node/examples) for the list and how to run them. diff --git a/content/docs/sdk/php/intro.mdx b/content/docs/sdk/php/intro.mdx index d29c108710..37857efe2d 100644 --- a/content/docs/sdk/php/intro.mdx +++ b/content/docs/sdk/php/intro.mdx @@ -34,25 +34,25 @@ On macOS the library is `libiggy_php.dylib`. ## Quick start -The sample below expects an Iggy server on `127.0.0.1:8090`: +The sample below targets Iggy 0.9.0 on `127.0.0.1:8090`. When testing unreleased SDK changes, build the server from the same source checkout: ```bash -# Latest release +# Server 0.9.0 docker run --rm \ --cap-add=SYS_NICE --security-opt seccomp=unconfined --ulimit memlock=-1:-1 \ -p 8090:8090 \ -e IGGY_TCP_ADDRESS=0.0.0.0:8090 \ -e IGGY_NODE_ADVERTISED_ADDRESS=localhost \ -e IGGY_ROOT_USERNAME=iggy -e IGGY_ROOT_PASSWORD=iggy \ - apache/iggy:latest + apache/iggy:0.9.0 -# Or from the repository source +# Or from the repository root cargo run --bin iggy-server -- --fresh --with-default-root-credentials ``` -The environment variables make the server reachable through the published port (it binds to `127.0.0.1` inside the container by default) and set the `iggy`/`iggy` root credentials the sample's connection string uses. +The environment variables make the server reachable through the published port (it binds to `127.0.0.1` inside the container by default) and bootstrap the `iggy`/`iggy` root credentials for a new data directory. Existing stored credentials are not replaced. For the source command, ensure `IGGY_ROOT_USERNAME` and `IGGY_ROOT_PASSWORD` are unset or match the sample; environment values take precedence over the flag. `--fresh` removes the local replica state, so use it only for disposable development data. -Credentials in the connection string take effect on `connect()`. Alternatively, construct the client with a plain address (`new \Iggy\Client('127.0.0.1:8090')`) and call `loginUser('iggy', 'iggy')` explicitly. +For binary transports, credentials in the connection string take effect on `connect()`. HTTP requires `loginUser()` explicitly. Alternatively, construct the client with a plain address (`new \Iggy\Client('127.0.0.1:8090')`) and call `loginUser('iggy', 'iggy')` explicitly. ```php consumerGroup( \Iggy\AutoCommit::disabled(), ); -// Callback style: the message limit is required and must be finite. +$limit = 5; $consumer->consumeMessages( function (\Iggy\ReceiveMessage $message) use ($consumer): void { - process($message->payload()); + echo $message->payload(), PHP_EOL; $consumer->storeOffset($message->offset(), $message->partitionId()); }, - 100, + $limit, ); -// Iterator style: break out when done. +$consumed = 0; foreach ($consumer->iterMessages() as $message) { - process($message->payload()); + echo $message->payload(), PHP_EOL; $consumer->storeOffset($message->offset(), $message->partitionId()); + if (++$consumed === $limit) { + break; + } } ``` @@ -129,7 +132,7 @@ foreach ($consumer->iterMessages() as $message) { - Partition ids are 0-based: for a topic with one partition, use `0`. - PHP strings are passed as named identifiers, including strings that contain only digits. PHP integers are passed as numeric identifiers. - `PollingStrategy::timestamp()` and `timestampMicros()` expect microseconds since the Unix epoch. Use `PollingStrategy::timestampSeconds()` for PHP `time()` values. -- Large unsigned values that can overflow PHP integers, such as message checksums, are returned as decimal strings. +- Message IDs and checksums are returned as decimal strings. Offset and timestamp getters return PHP integers and are limited to `PHP_INT_MAX`. - Errors are thrown as exceptions under `Iggy\Exception`: `AuthenticationException`, `ConnectionException`, `NotFoundException`, `TransientException`, and the `IggyException` base. - TLS uses the Rust SDK connection-string format, for example `iggy+tcp://user:pass@host:port?tls=true&tls_domain=localhost&tls_ca_file=/path/to/ca.pem`. - The extension owns a lazy global Tokio runtime. Don't call `pcntl_fork()` after the first Iggy call: the child inherits file descriptors but not the runtime's worker threads. @@ -140,9 +143,10 @@ foreach ($consumer->iterMessages() as $message) { ## Examples -Working examples are available in the [examples/php](https://github.com/apache/iggy/tree/master/examples/php) directory. They read `IGGY_CONNECTION_STRING`, or build one from `IGGY_HOST`, `IGGY_PORT`, `IGGY_USERNAME`, and `IGGY_PASSWORD`: +Working examples are available in the [examples/php](https://github.com/apache/iggy/tree/master/examples/php) directory. They read `IGGY_CONNECTION_STRING` for a binary transport with credentials. Otherwise they connect over TCP using `IGGY_HOST` and `IGGY_PORT`, then log in with the literal `IGGY_USERNAME` and `IGGY_PASSWORD` values: ```bash +# From the repository root cd examples/php (cd ../../foreign/php && cargo build) export PHP_IGGY_EXTENSION="$(pwd)/../../foreign/php/target/debug/libiggy_php.so" diff --git a/content/docs/sdk/python/examples.mdx b/content/docs/sdk/python/examples.mdx index 9bf7c1150f..7852bb2613 100644 --- a/content/docs/sdk/python/examples.mdx +++ b/content/docs/sdk/python/examples.mdx @@ -3,7 +3,7 @@ title: Examples description: "Working Python examples from the core repository, covering connection strings, message headers and TLS." --- -Working examples are available in the [examples/python](https://github.com/apache/iggy/tree/master/examples/python) directory. CI exercises them, so they always match the current SDK: +Working examples are available in the [examples/python](https://github.com/apache/iggy/tree/master/examples/python) directory. CI runs the source examples against a source-built server, including a separate TLS pass: - **getting-started** - producer and consumer with optional TLS via `TcpConfig` - **basic** - producer and consumer using connection strings @@ -11,25 +11,25 @@ Working examples are available in the [examples/python](https://github.com/apach ## Running the examples -Start an Iggy server first: +These examples target Iggy 0.9.0. For unreleased changes, use the server and Python SDK from the same source checkout: ```bash -# Latest release +# Server 0.9.0 docker run --rm \ --cap-add=SYS_NICE --security-opt seccomp=unconfined --ulimit memlock=-1:-1 \ -p 8090:8090 \ -e IGGY_TCP_ADDRESS=0.0.0.0:8090 \ -e IGGY_NODE_ADVERTISED_ADDRESS=localhost \ -e IGGY_ROOT_USERNAME=iggy -e IGGY_ROOT_PASSWORD=iggy \ - apache/iggy:latest + apache/iggy:0.9.0 -# Or from the repository source +# Or from the repository root cargo run --bin iggy-server -- --fresh --with-default-root-credentials ``` -The environment variables make the server reachable through the published port (it binds to `127.0.0.1` inside the container by default) and set the `iggy`/`iggy` root credentials the examples default to. +The environment variables make the server reachable through the published port (it binds to `127.0.0.1` inside the container by default) and bootstrap the `iggy`/`iggy` root credentials for new data. Stored credentials are not replaced, and environment credentials override the source command's default-credentials flag. Use `--fresh` only with disposable local replica data. -Then, from `examples/python`, install the dependencies and run a producer/consumer pair: +With Python 3.10 or newer and Rust/Cargo available, run from `examples/python`. `uv` uses the local SDK path in `pyproject.toml`; the pip command below names that path explicitly: ```bash # Using uv @@ -40,7 +40,7 @@ uv run getting-started/consumer.py # Without uv python -m venv .venv source .venv/bin/activate -pip install . +pip install ../../foreign/python . python getting-started/producer.py python getting-started/consumer.py ``` diff --git a/content/docs/sdk/python/intro.mdx b/content/docs/sdk/python/intro.mdx index d114ef2e84..9c19f80009 100644 --- a/content/docs/sdk/python/intro.mdx +++ b/content/docs/sdk/python/intro.mdx @@ -5,33 +5,43 @@ description: "The Python SDK, a PyO3 wrapper around the Rust SDK, and how to ins The Iggy Python SDK is a client library that allows you to interact with the Iggy API from your Python application. It is built as a PyO3 wrapper around the Rust SDK, which means it supports TCP, QUIC, HTTP, and WebSocket transports via connection strings. The package is available on [PyPI](https://pypi.org/project/apache-iggy/) and the source code can be found on [GitHub](https://github.com/apache/iggy/tree/master/foreign/python). -Because the wheel bundles the Rust SDK, it speaks only the current Iggy wire protocol and doesn't fall back to older formats. Keep the client and server versions in step: pair the newest wheel with the newest server. +The wheel bundles the Rust SDK, whose binary transports speak the current Iggy wire protocol without falling back to older formats. Use an SDK release compatible with your server. For unreleased changes, build the SDK and server from the same source checkout; the published wheel can lag source API changes. ## Installation +Python 3.10 or newer is required. Install the published package in a virtual environment: + ```bash +python -m venv .venv +source .venv/bin/activate pip install apache-iggy ``` +For the source version described here, install from the repository root with Rust and Cargo available: + +```bash +pip install ./foreign/python +``` + ## Quick start -The samples below expect an Iggy server on `127.0.0.1:8090`: +The samples below target Iggy 0.9.0 on `127.0.0.1:8090`: ```bash -# Latest release +# Server 0.9.0 docker run --rm \ --cap-add=SYS_NICE --security-opt seccomp=unconfined --ulimit memlock=-1:-1 \ -p 8090:8090 \ -e IGGY_TCP_ADDRESS=0.0.0.0:8090 \ -e IGGY_NODE_ADVERTISED_ADDRESS=localhost \ -e IGGY_ROOT_USERNAME=iggy -e IGGY_ROOT_PASSWORD=iggy \ - apache/iggy:latest + apache/iggy:0.9.0 -# Or from the repository source +# Or from the same repository root as the SDK cargo run --bin iggy-server -- --fresh --with-default-root-credentials ``` -The environment variables make the server reachable through the published port (it binds to `127.0.0.1` inside the container by default) and set the `iggy`/`iggy` root credentials the samples' connection string uses. Without `--with-default-root-credentials` (or the `IGGY_ROOT_USERNAME` / `IGGY_ROOT_PASSWORD` variables), a first boot generates a random root password instead. +The environment variables make the server reachable through the published port (it binds to `127.0.0.1` inside the container by default) and bootstrap the `iggy`/`iggy` root credentials for new data. Stored credentials are not replaced. Environment credentials override the default-credentials flag, so unset them or set them to match the sample. `--fresh` deletes local replica state; use disposable development data. A new standalone server without explicit credentials generates a root password; a new cluster requires explicit credentials. ### Producer @@ -82,7 +92,7 @@ asyncio.run(main()) ```python import asyncio -from apache_iggy import IggyClient, PollingStrategy +from apache_iggy import Consumer, IggyClient, PollingStrategy STREAM_NAME = "sample-stream" TOPIC_NAME = "sample-topic" @@ -95,11 +105,12 @@ async def main(): ) await client.connect() - # Next() with auto_commit=True continues from this consumer's last committed - # offset, so each run picks up where the previous one finished. + # Next() resumes this named consumer after its stored offset. Auto-commit + # can store the offset before application processing finishes. polled_messages = await client.poll_messages( stream=STREAM_NAME, topic=TOPIC_NAME, + consumer=Consumer.Single("sample-consumer"), partition_id=PARTITION_ID, polling_strategy=PollingStrategy.Next(), count=10, @@ -116,8 +127,8 @@ asyncio.run(main()) ## Beyond the basics -- **TLS**: construct the client from a `TcpConfig` instead of a connection string and set `tls_enabled` plus `tls_ca_file`. The [getting-started example](https://github.com/apache/iggy/tree/master/examples/python/getting-started) shows the full setup. -- **Consumer groups**: `client.consumer_group(...)` returns an `IggyConsumer` that creates and joins the group by default and commits offsets according to the configured `AutoCommit` mode. +- **TLS**: use `TcpConfig` with `tls_enabled` and `tls_ca_file`, or the TLS options in a [connection string](/docs/sdk/connection-strings). The certificate must match `tls_domain`, which defaults to the dialed host. The [getting-started example](https://github.com/apache/iggy/tree/master/examples/python/getting-started) shows the full setup. +- **Consumer groups**: `await client.consumer_group(...)` returns an `IggyConsumer` that creates and joins the group by default and commits offsets according to the configured `AutoCommit` mode. Groups require a binary transport; HTTP supports ordinary consumers only. - **User headers**: `SendMessage(data, user_headers=...)` accepts a plain `dict` with `str`, `bytes`, `bool`, `int`, or `float` values. `HeaderKey`/`HeaderValue` give explicit control over the wire type. See the [message-headers examples](https://github.com/apache/iggy/tree/master/examples/python/message-headers). - **Topic options**: `create_topic(..., options=...)` accepts extra option keys as a `dict[str, str]`, validated against the server's catalog. `client.describe_options("topic")` lists the keys a server accepts. - **Administration**: user and permission management. Personal access tokens can be used for login via `AutoLogin.personal_access_token(...)` in the connection config. PAT management isn't exposed yet. diff --git a/content/docs/sdk/rust/examples.mdx b/content/docs/sdk/rust/examples.mdx index b3d89278d3..0e0617bf35 100644 --- a/content/docs/sdk/rust/examples.mdx +++ b/content/docs/sdk/rust/examples.mdx @@ -5,13 +5,15 @@ description: "The Rust SDK examples in the core repository, and how to start a s In the core repository, you can find the following [examples](https://github.com/apache/iggy/tree/master/examples/rust/src) using the Rust SDK. -Before running any of them, start the server with the default root credentials. Without the flag (or `IGGY_ROOT_USERNAME`/`IGGY_ROOT_PASSWORD`) root gets a **generated password** and every example fails with `InvalidCredentials`: +Build the SDK and server from the same checkout for unreleased changes. Start the server from the repository root in a separate terminal. The examples use `iggy`/`iggy`; for a new local instance with no `IGGY_ROOT_USERNAME` or `IGGY_ROOT_PASSWORD` overrides: ```bash cargo run --bin iggy-server -- --fresh --with-default-root-credentials ``` -The examples are cargo example targets of the `iggy_examples` crate. Run them from the `examples/rust` directory with `cargo run --example `. Partition IDs are **0-based**: the examples send to and poll from partition `0`. +Environment credentials take precedence over the flag, and bootstrap settings do not replace recovered credentials. `--fresh` deletes this replica's local data; use disposable development data. A new standalone server without explicit credentials generates a root password; a new cluster requires explicit credentials. + +The examples are cargo example targets of the `iggy_examples` crate. Run them from the **repository root** with `cargo run --example ` so relative certificate paths resolve. Run each producer before its consumer. Some examples create fixed names and require fresh data when repeated; the getting-started consumer also expects stream and topic IDs `0`, so run that pair first on a fresh server. Partition IDs are **0-based**, but balanced producers and consumer groups can use multiple partitions. - **[Getting started](https://github.com/apache/iggy/tree/master/examples/rust/src/getting-started)** - the basic example which is discussed in the [getting started](/docs/introduction/getting-started) guide. @@ -69,7 +71,16 @@ The examples are cargo example targets of the `iggy_examples` crate. Run them fr cargo run --example multi-tenant-consumer ``` -- **[TCP TLS](https://github.com/apache/iggy/tree/master/examples/rust/src/tcp-tls)** - TLS-encrypted TCP connections with custom CA certificates. +- **[TCP TLS](https://github.com/apache/iggy/tree/master/examples/rust/src/tcp-tls)** - TLS-encrypted TCP connections with custom CA certificates. Stop the plain server and start a TLS-enabled server from the repository root: + + ```bash + IGGY_TCP_TLS_ENABLED=true \ + IGGY_TCP_TLS_CERT_FILE=core/certs/iggy_cert.pem \ + IGGY_TCP_TLS_KEY_FILE=core/certs/iggy_key.pem \ + cargo run --bin iggy-server -- --fresh --with-default-root-credentials + ``` + + These certificates are for local testing. Then run: ```bash cargo run --example tcp-tls-producer diff --git a/content/docs/sdk/rust/high-level-sdk.mdx b/content/docs/sdk/rust/high-level-sdk.mdx index 9ed00c0398..f4b90347b9 100644 --- a/content/docs/sdk/rust/high-level-sdk.mdx +++ b/content/docs/sdk/rust/high-level-sdk.mdx @@ -6,41 +6,50 @@ description: "IggyProducer and IggyConsumer: batching, consumer groups and offse If you've read through the [getting started](/docs/introduction/getting-started) guide, you might have noticed that it's quite verbose and requires a lot of boilerplate code to get started. This is where the High-level SDK comes in, as it does provide a more user-friendly interface to interact with the Iggy API for both, producer and consumer. Let's consider the following features: - **Automatically creating & joining** the consumer groups -- **Committing the offset** depending on the particular mode (e.g. in the background based on some interval, after polling N messages etc.) +- **Committing the offset** depending on the particular mode (e.g. in the background based on some interval, when an offset matches the configured trigger etc.) - **Batching the messages**, whether it's about producing or consuming - **Processing the messages** as if the stream was an async iterator - **Reusing the same client** for both, producing and consuming on the same topic without repeating the configuration - And more... +The Rust snippets assume an async function returning `Result<(), Box>`, with `iggy`, Tokio and `futures-util` dependencies. Use the [Rust SDK setup](/docs/sdk/rust/intro) and a server with the sample credentials. + ## Connection string Instead of providing the configuration for the client, you can use the connection string. It's a string that contains all the necessary information to connect to the Iggy API, and it works with all four transports: -```bash -iggy://iggy:iggy@localhost:8090 (TCP, default) -iggy+tcp://iggy:iggy@localhost:8090 (TCP, explicit) -iggy+quic://iggy:iggy@localhost:8080 (QUIC) -iggy+http://iggy:iggy@localhost:3000 (HTTP) -iggy+ws://iggy:iggy@localhost:8092 (WebSocket) +``` +iggy://iggy:iggy@127.0.0.1:8090 (TCP, default) +iggy+tcp://iggy:iggy@127.0.0.1:8090 (TCP, explicit) +iggy+quic://iggy:iggy@127.0.0.1:8080 (QUIC) +iggy+http://iggy:iggy@127.0.0.1:3000 (HTTP) +iggy+ws://iggy:iggy@127.0.0.1:8092 (WebSocket) ``` Which can be used to create the client like this: ```rust -let client = IggyClient::from_connection_string("iggy://iggy:iggy@localhost:8090")?; +use futures_util::StreamExt; +use iggy::prelude::*; +use std::str::FromStr; + +let client = IggyClient::from_connection_string("iggy://iggy:iggy@127.0.0.1:8090")?; +client.connect().await?; ``` +HTTP requires an explicit login after construction. Consumer groups require a binary transport; use an ordinary consumer with an explicit partition over HTTP. + Options are appended as query parameters. For TCP: -```bash -iggy://iggy:iggy@localhost:8090?tls=true&tls_domain=test.com&reconnection_retries=5&reconnection_interval=5s&reestablish_after=10s&heartbeat_interval=3s&nodelay=true +``` +iggy://iggy:iggy@127.0.0.1:8090?reconnection_retries=5&reconnection_interval=5s&reestablish_after=10s&heartbeat_interval=3s&nodelay=true ``` The option keys differ per transport (for example QUIC uses `reconnection_max_retries` and `reconnection_reestablish_after` where TCP uses `reconnection_retries` and `reestablish_after`). Unknown keys are rejected as hard errors. See [Connection Strings](/docs/sdk/connection-strings) for the full per-transport option tables and defaults. -Unless you need to provide a specific implementation of the client-side `Encryptor`, `Partitioner` or adjust some other settings, you should be good to go with the connection string. On the other hand, you can always make use of `IggyClientBuilder::from_connection_string()` to extend the options on top of the provided connection string. +Unless you need client-side encryption, a custom `Partitioner`, or other settings outside the connection-string options, you should be good to go with the connection string. On the other hand, you can always make use of `IggyClientBuilder::from_connection_string()` to extend the options on top of the provided connection string. -The `IggyClient` internally wraps its transport in an `IggyRwLock`, so a single instance is **cheap to clone** and can be shared across multiple tasks. Typically, it's a good idea to create a separate connection for producing and consuming, but it's not a requirement. +The `IggyClient` internally wraps its transport in an `IggyRwLock`, but the client itself does not implement `Clone`. Share an `Arc` across tasks; cloning the `Arc` keeps the same client and connection. Typically, it's a good idea to create a separate connection for producing and consuming, but it's not a requirement. ## Producer @@ -60,7 +69,7 @@ let producer = client producer.init().await?; ``` -The code above will result in creating the producer that will try to send the messages in batches of 1000 every 5 milliseconds. You can choose between the `direct` (an instant producer) or the `background` (which will send the messages in the background by buffering them). The partitioning is set to `balanced` which means that the producer will try to distribute the messages evenly across all the partitions. The `init()` method is used to ensure that the producer is ready to send the messages by validating the existence of the stream, topic etc. +The direct producer splits each `send()` call into requests of at most 1000 messages. For sequential calls, the 5 ms linger setting delays a call until at least 5 ms after the previous successful send; it does not buffer messages between calls or delay individual chunks within a call. Choose `background` to buffer sends on worker tasks. `balanced` routes requests round-robin across partitions. By default, `init()` creates a missing stream and topic, requiring the corresponding permissions. Finally, you can use the `send()` method to send the messages to the topic. The producer **never needs to be a mutable binding**: both `init()` and `send()` take `&self`, so a plain `let producer` is enough for its whole life. Here's how you can send the messages: @@ -80,22 +89,22 @@ The `background` mode buffers messages and sends them from shard workers, tradin | Field | Default | Description | |-------|---------|-------------| | `num_shards` | `1` | Number of shard workers running in parallel | -| `sharding` | `OrderedSharding` | Maps a message to a shard. `OrderedSharding` routes the same stream/topic to the same shard, preserving order. `BalancedSharding` round-robins across shards for maximum throughput, order not preserved | +| `sharding` | `OrderedSharding` | Routes each send to a worker. `OrderedSharding` keeps a stream/topic pair on one sequential worker; `BalancedSharding` round-robins across workers for maximum throughput and can reorder sends | | `linger_time` | `1ms` | How long a shard may wait before flushing an incomplete batch | -| `batch_size` | 1 MiB | Maximum total **size in bytes** of a batch; `0` disables size-based batching | -| `batch_length` | `1000` | Maximum **number of messages** per batch; `0` disables length-based batching | -| `failure_mode` | `Block` | `BackpressureMode` applied when limits are reached: `Block`, `BlockWithTimeout(duration)`, or `FailImmediately` | -| `max_buffer_size` | 32 MiB | Upper bound for bytes held in memory across all shards; `0` means unlimited | -| `max_in_flight` | `1` | Maximum batches being sent concurrently. More than 1 can reorder messages when retries occur; `0` means unlimited | +| `batch_size` | 1 MiB | Per-worker flush threshold in bytes; `0` disables this trigger | +| `batch_length` | `1000` | Per-worker flush threshold in **queued sends**, not individual messages; `0` disables this trigger | +| `failure_mode` | `Block` | Behavior when the byte budget is exhausted: `Block`, `BlockWithTimeout(duration)`, or `FailImmediately` | +| `max_buffer_size` | 32 MiB | Budget for message bytes buffered or in flight across all workers; `0` disables this budget | +| `max_in_flight` | `1` | Maximum concurrent writes across workers; each worker remains sequential. `0` uses the semaphore's maximum capacity | | `error_callback` | `LogErrorCallback` | Async callback invoked on errors the producer cannot recover from | -`batch_size` limits bytes while `batch_length` limits the message count. Whichever limit fires first (including `linger_time`) triggers the flush. +`batch_size` and `batch_length` are flush triggers checked after adding a send. The final send can take the buffer past the byte threshold. The linger deadline starts when an empty buffer receives a send; the first trigger reached causes a flush. Each worker also has a bounded queue, which can make dispatch wait independently of `failure_mode`. A send larger than the entire byte budget fails under every backpressure mode. Nonzero `max_buffer_size` and `max_in_flight` values must not exceed `tokio::sync::Semaphore::MAX_PERMITS`; constructing the dispatcher with a larger value panics. -Call `producer.shutdown().await` to flush the background buffer before exiting. Dropping the producer without it **silently discards unflushed messages**. +Stop all senders, then call `producer.shutdown().await` to drain background queues and wait for writes and error callbacks. Dropping the producer without shutdown can lose buffered messages. Shutdown does not turn failed writes into successful ones; the default error callback logs and drops failed sends. ### Send confirmations -Every send method returns a `SendMessagesResponse` carrying the commit confirmations of the chunks the send was split into. The list is **empty** whenever the server sends no confirmation payload, and always for a `background` producer, which hands the messages to a dispatcher and returns before the send happens. Branch on `confirmations.is_empty()` instead of indexing. Delivery is *at-least-once*: a retried chunk may already have been committed by an earlier attempt, so an offset in a confirmation never implies uniqueness. +Every send method returns a `SendMessagesResponse` carrying the commit confirmations of the chunks the send was split into. The list is **empty** whenever the server sends no confirmation payload, and always for a `background` producer, which hands the messages to a dispatcher and returns before the send happens. Branch on `confirmations.is_empty()` instead of indexing. A retried chunk may already have been committed by an earlier attempt, so a confirmation offset does not imply uniqueness. Direct-send completion follows the topic's [durability policy](/docs/server/durability). ## Consumer @@ -105,7 +114,7 @@ The consumer is a high-level abstraction that allows you to receive the messages let mut consumer = client .consumer_group("my-consumer-group", "my-stream", "my-topic")? .auto_commit(AutoCommit::IntervalOrWhen( - IggyDuration::from_str("1s")?, + NonZeroIggyDuration::ONE_SECOND, AutoCommitWhen::ConsumingAllMessages, )) .create_consumer_group_if_not_exists() @@ -116,7 +125,7 @@ let mut consumer = client .build(); ``` -The code above will result in creating the consumer that will try to consume the messages in batches of 1000 every 1 millisecond. The auto-commit is set to commit the offset every second or when all the messages are consumed (fetched). The polling strategy is set to `next` which means that the consumer will try to consume the next available message from the partition currently assigned to the consumer group (you can also invoke a regular `consumer()` builder if you do not plan to use the consumer groups). The `build()` method is used to create the consumer. +The consumer polls at most 1000 messages per request with a minimum 1 ms gap between polls. It requests offset commits every second or when the current message buffer becomes empty; these triggers do not observe application processing. The polling strategy is set to `next` which means that the consumer will try to consume the next available message from the partition currently assigned to the consumer group (you can also invoke a regular `consumer()` builder if you do not plan to use the consumer groups). The `build()` method is used to create the consumer. Finally, you can use the `next()` method to receive the messages from the topic. Unlike the producer, the consumer **must stay a mutable binding** for its whole life: `init()` takes `&mut self`, and `next()` comes from the futures `Stream` implementation which also requires mutable access. The `init()` is used to ensure that the consumer is ready to receive the messages by validating the existence of the stream, topic, consumer group etc. Here's how you can consume the messages: @@ -127,7 +136,7 @@ consumer.init().await?; while let Some(message) = consumer.next().await { match message { Ok(received) => { - // received.message is the IggyMessage + println!("{}", String::from_utf8_lossy(&received.message.payload)); } Err(error) => eprintln!("Error while receiving message: {error}"), } @@ -152,14 +161,15 @@ Defaults and lesser-known knobs on `IggyConsumerBuilder`: | `init_retries(retries, interval)` | disabled | Retry `init()` when the stream or topic does not exist yet (e.g. created dynamically by a producer) | | `allow_replay()` | disabled | Allow re-consuming messages at or below the stored offset | | `offset_drain_timeout(d)` | `5s` | How long `shutdown()` waits for background auto-commit tasks to drain before leaving the group | -| `commit_failed_messages()` | not set | Sets `AutoCommit::Disabled`, so offsets of failed messages are never skipped; store offsets manually after successful processing | -| `encryptor(...)` | none | Client-side payload decryption | +| `encryptor(...)` | inherited from the client; none by default | Client-side payload and user-header decryption | + +With an encryptor, `init()` rejects the `PollingMessages` auto-commit trigger because a batch might fail decryption after its offset was committed. Use a compatible mode, such as `AutoCommit::Disabled`, and store offsets explicitly after successful processing. ### Auto-commit matrix -`AutoCommit` decides when the consumer offset is stored on the server: +`AutoCommit` decides when the consumer requests offset storage: -| Variant | Offset is stored | +| Variant | Commit trigger | |---------|------------------| | `Disabled` | Never automatically; store it manually | | `Interval(d)` | Every `d` in the background | @@ -168,13 +178,15 @@ Defaults and lesser-known knobs on `IggyConsumerBuilder`: | `When(when)` | At the `when` trigger | | `After(after)` | At the `after` trigger | -`AutoCommitWhen` triggers fire while receiving: `PollingMessages`, `ConsumingAllMessages`, `ConsumingEachMessage`, `ConsumingEveryNthMessage(n)`. `AutoCommitAfter` triggers fire after processing: `ConsumingAllMessages`, `ConsumingEachMessage`, `ConsumingEveryNthMessage(n)`. +`AutoCommitWhen` triggers fire while receiving: `PollingMessages` accompanies the poll request, `ConsumingEachMessage` runs before yielding a message, and `ConsumingAllMessages` runs when the current buffer is empty. `ConsumingEveryNthMessage(n)` checks whether the message offset is divisible by `n`, rather than counting processed messages; `n = 0` disables that trigger. + +`AutoCommitAfter` triggers run after the handler returns, including when it returns an error. `ConsumingEachMessage` and `ConsumingEveryNthMessage(n)` use the handled message's offset. `ConsumingAllMessages` fires when that offset reaches the partition head reported by the poll, so a lagging consumer may process several batches before it fires. For commits only after successful processing, disable auto-commit and store offsets explicitly. -The `After` and `IntervalOrAfter` variants *only* work with `consume_messages()` from the `IggyConsumerMessageExt` trait (see [Stream Builder](/docs/sdk/rust/stream-builder)). The plain `next()` loop cannot observe when your processing finished. +The `After` triggers require `consume_messages()` from the `IggyConsumerMessageExt` trait (see [Stream Builder](/docs/sdk/rust/stream-builder)). The plain `next()` loop cannot observe when processing finished, though `IntervalOrAfter` still commits on its interval. Background commits can fail; use an explicit offset-store result when completion must be confirmed. ## Raw requests For custom commands the `IggyClient` exposes two escape hatches below the typed API: -- `send_binary_request(code, payload)` sends a raw binary command and returns the raw response. Binary transports only. On HTTP it returns `FeatureUnavailable`. Login and logout codes are rejected with `InvalidCommand`, use `login_user`/`logout_user` so the SDK session state stays correct. Custom codes are forwarded to the server, which decides whether it implements them. +- `send_binary_request(code, payload)` sends a raw binary command and returns the raw response. Binary transports only. On HTTP it returns `FeatureUnavailable`. Login, registration and logout codes are rejected with `InvalidCommand`, use `login_user`/`logout_user` so the SDK session state stays correct. Custom codes are forwarded to the server, which decides whether it implements them. - `send_http_request(method, path, body)` invokes an arbitrary HTTP endpoint and returns the raw response body. HTTP transport only. Binary transports return `FeatureUnavailable`. diff --git a/content/docs/sdk/rust/intro.mdx b/content/docs/sdk/rust/intro.mdx index 20e46f7ae4..7bb98cd961 100644 --- a/content/docs/sdk/rust/intro.mdx +++ b/content/docs/sdk/rust/intro.mdx @@ -5,23 +5,27 @@ description: "The Rust SDK, the primary and most complete Iggy client, and where The Rust SDK is the primary and most feature-complete client for Iggy. It is available on [crates.io](https://crates.io/crates/iggy) and the source code is part of the [core repository](https://github.com/apache/iggy/tree/master/core/sdk). +Run from your application crate to add the published SDK: + ```bash cargo add iggy ``` +Use an SDK release compatible with your server. For unreleased changes, build both from the same checkout; the [Getting Started](/docs/introduction/getting-started) guide shows the source dependency setup. + ## Prerequisites -The examples below log in with the `iggy`/`iggy` root credentials, which **only exist** when the server is started with the default-credentials flag (or `IGGY_ROOT_USERNAME`/`IGGY_ROOT_PASSWORD` set) on its first boot. Otherwise root gets a generated password and the examples fail with `InvalidCredentials`: +The examples below log in with `iggy`/`iggy`. For a new local development instance, with no `IGGY_ROOT_USERNAME` or `IGGY_ROOT_PASSWORD` overrides, use: ```bash cargo run --bin iggy-server -- --fresh --with-default-root-credentials ``` -`--fresh` **wipes the local data directory**. This setup is for *development only*. +`--fresh` **wipes this replica's local data directory**. Environment credentials take precedence over the flag, and bootstrap settings do not replace recovered credentials. In a cluster, a fresh replica can recover credentials from peers. Use disposable development data. A new standalone server without explicit credentials generates a root password; a new cluster requires explicit credentials. ## High-level vs low-level API -The SDK provides two layers: +The SDK provides two layers. The snippets below assume an async Tokio context with a `Result` return type. **High-level API (recommended)** - the easiest way to get started. It handles connection management, auto-batching, consumer group lifecycle, offset commits, retry logic, and reconnection out of the box. Use `IggyClient`, `IggyProducer`, and `IggyConsumer` for the best developer experience. @@ -29,13 +33,13 @@ The SDK provides two layers: use iggy::prelude::*; // Connect with a connection string -let client = IggyClient::from_connection_string("iggy://iggy:iggy@localhost:8090")?; +let client = IggyClient::from_connection_string("iggy://iggy:iggy@127.0.0.1:8090")?; client.connect().await?; // Or use the builder let client = IggyClientBuilder::new() .with_tcp() - .with_server_address("localhost:8090".to_string()) + .with_server_address("127.0.0.1:8090".to_string()) .build()?; client.connect().await?; client.login_user("iggy", "iggy").await?; @@ -57,17 +61,19 @@ iggy+http://user:pass@host:port (HTTP) iggy+ws://user:pass@host:port (WebSocket) ``` +TCP, QUIC, and WebSocket apply connection-string credentials on `connect()`. HTTP requires an explicit login and supports ordinary consumers only. + Options can be appended as query parameters: ``` -iggy://iggy:iggy@my-server:8090?tls=true&tls_ca_file=/path/to/ca.crt -iggy://iggy:iggy@localhost:8090?reconnection_retries=unlimited&heartbeat_interval=5s +iggy://iggy:iggy@127.0.0.1:8090?nodelay=true&heartbeat_interval=5s +iggy://iggy:iggy@127.0.0.1:8090?reconnection_retries=unlimited&heartbeat_interval=5s ``` Personal Access Tokens are also supported. Any credential without a colon is treated as a PAT. Paste the token exactly as the server returned it, with **no prefix**: ``` -iggy://@localhost:8090 +iggy://@127.0.0.1:8090 ``` See [Connection Strings](/docs/sdk/connection-strings) for the full per-transport option tables and defaults. @@ -138,6 +144,6 @@ async fn main() -> Result<(), Box> { } ``` -Partition IDs are **0-based**: the first partition of a topic is partition `0`. +Partition IDs are **0-based**: the first partition of a topic is partition `0`. The consumer uses the default consumer identity and requests auto-commit; its offset can advance before application processing finishes. Use explicit offset storage after successful processing when that distinction matters. For the full getting started tutorial, see [Getting Started](/docs/introduction/getting-started). For the high-level producer/consumer builders, see [High-level SDK](/docs/sdk/rust/high-level-sdk) and [Stream Builder](/docs/sdk/rust/stream-builder). diff --git a/content/docs/sdk/rust/stream-builder.mdx b/content/docs/sdk/rust/stream-builder.mdx index cfcf905fe3..697b45eb26 100644 --- a/content/docs/sdk/rust/stream-builder.mdx +++ b/content/docs/sdk/rust/stream-builder.mdx @@ -17,13 +17,15 @@ The stream builder provides a convenient way to create the iggy client, producer All source code examples are located in the [**examples folder**](https://github.com/apache/iggy/tree/master/examples/rust/src/stream-builder) of the iggy repository. Also, if you encounter a problem with any of the examples below, please ask in the [**community discord**](https://discord.gg/apache-iggy). -The examples below connect with the `iggy`/`iggy` root credentials, which **only exist** when the server was started with: +The examples below use `iggy`/`iggy`. For a new local development instance, with no `IGGY_ROOT_USERNAME` or `IGGY_ROOT_PASSWORD` overrides, use: ```bash cargo run --bin iggy-server -- --fresh --with-default-root-credentials ``` -Setting `IGGY_ROOT_USERNAME`/`IGGY_ROOT_PASSWORD` works too. Otherwise root gets a generated password and the examples fail with `InvalidCredentials`. +Environment credentials take precedence over the flag, and bootstrap settings do not replace recovered credentials. `--fresh` deletes this replica's local data; use disposable development data. A new standalone server without explicit credentials generates a root password; a new cluster requires explicit credentials. + +Use the SDK and server from the same checkout for unreleased changes. The examples import `PrintEventConsumer` from the local `iggy_examples` crate and can run as the targets in `examples/rust`. In your own application, use the implementation shown below instead of that import. ## IggyStream Builder @@ -35,7 +37,7 @@ use iggy_examples::shared::stream::PrintEventConsumer; use std::str::FromStr; use tokio::sync::oneshot; -const IGGY_URL: &str = "iggy://iggy:iggy@localhost:8090"; +const IGGY_URL: &str = "iggy://iggy:iggy@127.0.0.1:8090"; #[tokio::main] async fn main() -> Result<(), IggyError> { @@ -110,7 +112,7 @@ When you implement the producer side, you can use the `IggyStreamProducer` to ge use iggy::prelude::*; use std::str::FromStr; -const IGGY_URL: &str = "iggy://iggy:iggy@localhost:8090"; +const IGGY_URL: &str = "iggy://iggy:iggy@127.0.0.1:8090"; #[tokio::main] async fn main() -> Result<(), IggyError> { @@ -153,7 +155,7 @@ repository. The IggyProducerConfig gives you a way to configure the producer in sufficient detail. Please note, if you have questions about any of those settings, please ask in the community discord. For basic customization, the `from_stream_topic` constructor -lets you set a custom stream and topic name as well as the maximum batch length and linger time between sends. +lets you set a custom stream and topic name as well as the maximum batch length and linger time between sends. The producer resolves destinations by name; keep the corresponding identifier fields consistent with those names. ```rust use iggy::prelude::*; @@ -209,8 +211,8 @@ async fn main() -> Result<(), IggyError> { // - `PartitionId` - the partition ID is provided by the client. // - `MessagesKey` - the client hashes the provided messages key // to a partition ID. - // The partition count is cached per topic and not refreshed - // while the client stays connected. + // Binary clients cache the partition count for their lifetime; + // reconnecting does not refresh it. .partitioning(Partitioning::balanced()) // Sets the retry policy (maximum number of retries and interval) // in case of messages sending failure. @@ -218,12 +220,12 @@ async fn main() -> Result<(), IggyError> { // or to the server rejecting the messages. // Default is 3 retries with 1 second interval between them. .send_retries_count(3) - .send_retries_interval(IggyDuration::new_from_secs(1)) + .send_retries_interval(NonZeroIggyDuration::ONE_SECOND) // Optionally, set a custom client side encryptor for encrypting // the messages' payloads. Currently only Aes256Gcm is supported. // Note, this is independent of server side encryption. // You can add client encryption, server encryption, or both. - // .encryptor(Arc::new(EncryptorKind::Aes256Gcm(Aes256GcmEncryptor::new(&[1; 32])?))) + // .encryptor(std::sync::Arc::new(EncryptorKind::Aes256Gcm(Aes256GcmEncryptor::new(&[1; 32])?))) .build(); Ok(()) } @@ -245,7 +247,7 @@ use iggy::prelude::*; use iggy_examples::shared::stream::PrintEventConsumer; use tokio::sync::oneshot; -const IGGY_URL: &str = "iggy://iggy:iggy@localhost:8090"; +const IGGY_URL: &str = "iggy://iggy:iggy@127.0.0.1:8090"; #[tokio::main] async fn main() -> Result<(), IggyError> { @@ -322,27 +324,27 @@ async fn main() -> Result<(), IggyError> { .create_stream_if_not_exists(true) // Create the topic if it doesn't exist. .create_topic_if_not_exists(true) - // The name of the consumer. Must be unique. + // Members of the same consumer group use the same name. .consumer_name("test_consumer".to_string()) // The type of consumer. It can be either `Consumer` or `ConsumerGroup`. // ConsumerGroup is default. .consumer_kind(ConsumerKind::ConsumerGroup) - // Sets the number of partitions for ConsumerKind `Consumer`. - // Does not apply to `ConsumerGroup`. + // Partition count when creating a topic. Also used as the partition + // ID for an ordinary consumer; group assignment ignores this value. .partitions_count(1) // The polling interval for messages. .polling_interval(IggyDuration::from_str("5ms").unwrap()) // `PollingStrategy` specifies from where to start polling messages. // It has the following kinds: // - `Offset` - start polling from the specified offset. - // - `Timestamp` - start polling from the specified timestamp. + // - `Timestamp` - start at or after the broker append timestamp. // - `First` - start polling from the first message in the partition. - // - `Last` - start polling from the last message in the partition. + // - `Last` - read up to batch_length messages ending at the committed offset. // - `Next` - start polling from the next message after the // last polled message based on the stored consumer offset. .polling_strategy(PollingStrategy::last()) // Sets the polling retry interval in case of server disconnection. - .polling_retry_interval(IggyDuration::new_from_secs(1)) + .polling_retry_interval(NonZeroIggyDuration::ONE_SECOND) // Sets the number of retries and the interval when initializing // the consumer if the stream or topic is not found. // Useful when the stream or topic is created dynamically @@ -350,20 +352,22 @@ async fn main() -> Result<(), IggyError> { // The default is 5 retries with a 3 second interval, // so init retry is enabled out of the box. .init_retries(5) - .init_interval(IggyDuration::new_from_secs(1)) - // Optionally, set a custom client side encryptor for encrypting - // the messages' payloads. Currently only Aes256Gcm is supported. + .init_interval(NonZeroIggyDuration::ONE_SECOND) + // Optionally, configure Aes256Gcm payload and user-header decryption. + // Replace PollingMessages auto-commit with a compatible mode first. // Key must be identical to the one used by the producer; // thus ensure secure key exchange. // Note, this is independent of server side encryption. // you can add client encryption, server encryption, or both. - // .encryptor(Arc::new(EncryptorKind::Aes256Gcm(Aes256GcmEncryptor::new(&[1; 32])?))) + // .encryptor(std::sync::Arc::new(EncryptorKind::Aes256Gcm(Aes256GcmEncryptor::new(&[1; 32])?))) .build(); Ok(()) } ``` +`partitions_count` also selects the partition ID for an ordinary consumer. When it creates a topic with `n` partitions, the valid IDs are `0..n-1`, so using the same `n` as an ordinary consumer's ID fails. The sample uses a consumer group. For an ordinary consumer, create the topic separately and use `client.consumer(name, stream, topic, partition_id)` to select a valid partition. + ## Add consumers dynamically at runtime. When you create consumers on demand at application runtime, the consumer has to survive the window where the stream @@ -376,13 +380,11 @@ you have to set `create_stream_if_not_exists` and `create_topic_if_not_exists` t the `IggyStreamConsumer` constructors as before i.e.: ```rust - let config = get_my_custom_iggy_consumer_config(); let (client, mut consumer) = IggyStreamConsumer::with_client_from_url(IGGY_URL, &config).await?; ``` -Where `get_my_custom_iggy_consumer_config` refers to a function that returns an `IggyConsumerConfig` -that specifies the stream and topic to consume as well the init retry or whether to create the stream and topic. +Here `config` is the `IggyConsumerConfig` built above, which selects the stream and topic, initialization retries, and automatic creation behavior. ## Add producers dynamically at runtime. From 93d1cd4427af3a5ecf37617f3f9b8df45d699082 Mon Sep 17 00:00:00 2001 From: hubcio Date: Fri, 11 Sep 2026 16:20:42 +0200 Subject: [PATCH 09/13] fix(docs): clarify telemetry availability --- content/docs/ai/mcp.mdx | 2 ++ content/docs/introduction/about.mdx | 2 +- content/docs/server/configuration.mdx | 2 ++ 3 files changed, 5 insertions(+), 1 deletion(-) diff --git a/content/docs/ai/mcp.mdx b/content/docs/ai/mcp.mdx index 201150e7ef..22d3f4e643 100644 --- a/content/docs/ai/mcp.mdx +++ b/content/docs/ai/mcp.mdx @@ -74,6 +74,8 @@ transport = "grpc" # grpc or http endpoint = "http://localhost:4317" ``` +Set `telemetry.enabled = true` to export to a running collector. For HTTP export, set `transport = "http"` and use complete signal URLs: `http://localhost:4318/v1/logs` for logs and `http://localhost:4318/v1/traces` for traces. The MCP server does not append those paths. + The configuration file must be in the `toml` format. By default, the server looks for `core/ai/mcp/config.toml` relative to its working directory. Set `IGGY_MCP_CONFIG_PATH` to use another path. Embedded defaults are loaded first, then the file if it exists, then environment overrides. Each setting can also be overridden using `IGGY_MCP_