From 8940cff1e30ecf97b7abbcf21c9a069d24f60942 Mon Sep 17 00:00:00 2001 From: FujitsuPolycom <87842395+FujitsuPolycom@users.noreply.github.com> Date: Sun, 6 Sep 2026 17:43:56 -0500 Subject: [PATCH 1/5] Attribute request cache reuse at verified scheduler boundaries Emit optional admission, restore-finalization, accepted prompt-work, preemption and terminal events for the SparkCache request ledger. Preserve deferred GPU lease attribution and dispatch ranges across asynchronous scheduling. Apply an exact-source transform after verifying checkpoint ownership dependencies; retain immutable payloads and published receipts. Validation: 470 mesh tests passed, four optional skips, including five companion-ledger integration tests; Ruff and documentation links passed. Hardware validation requires a rebuilt image. --- .../performance/README.md | 9 + .../performance/attribution/README.md | 81 +++++++ .../attribution/patch_scheduler.py | 127 ++++++++++ .../attribution/test_connector_integration.py | 186 +++++++++++++++ .../attribution/test_patch_scheduler.py | 223 ++++++++++++++++++ .../performance/install.py | 11 + .../performance/prepare.py | 2 +- 7 files changed, 638 insertions(+), 1 deletion(-) create mode 100644 runtime/glm53-spark-mtp3-mesh/performance/attribution/README.md create mode 100644 runtime/glm53-spark-mtp3-mesh/performance/attribution/patch_scheduler.py create mode 100644 runtime/glm53-spark-mtp3-mesh/performance/attribution/test_connector_integration.py create mode 100644 runtime/glm53-spark-mtp3-mesh/performance/attribution/test_patch_scheduler.py diff --git a/runtime/glm53-spark-mtp3-mesh/performance/README.md b/runtime/glm53-spark-mtp3-mesh/performance/README.md index 9d49f221..8ce99a8f 100644 --- a/runtime/glm53-spark-mtp3-mesh/performance/README.md +++ b/runtime/glm53-spark-mtp3-mesh/performance/README.md @@ -84,3 +84,12 @@ The build context includes the wrapper, warmup client, and admission module; the image receipt verifies all three files. Published image receipts describe immutable artifacts and do not claim this behavior until an image containing these sources has been built and recorded. + +## Request reuse accounting + +The image build includes [scheduler attribution hooks](attribution/README.md) +for the connector's opt-in request ledger. These hooks distinguish admitted +local reuse, finalized persistent restoration, and accepted prompt work across +preemption attempts. They are inactive unless the connector enables request +cache events. Source availability does not change a published image receipt; +a rebuilt image must be validated before serving evidence is claimed. diff --git a/runtime/glm53-spark-mtp3-mesh/performance/attribution/README.md b/runtime/glm53-spark-mtp3-mesh/performance/attribution/README.md new file mode 100644 index 00000000..7951432a --- /dev/null +++ b/runtime/glm53-spark-mtp3-mesh/performance/attribution/README.md @@ -0,0 +1,81 @@ +# Request cache attribution at scheduler boundaries + +Status: **implemented** with CPU scheduler-seam coverage. The MTP3 profile +remains **research-only**; rebuilt-image validation is required. + +`patch_scheduler.py` applies a byte-checked transform after the recurrent +checkpoint payload has been installed. The payload files and their manifest +are immutable inputs. The image installer first verifies every checkpoint +ownership dependency, applies the transform, and changes only the scheduler +entry to the transform's expected output hash. The image receipt records both +transform hashes and verifies the installed scheduler file. + +## Connector interface + +Instrumentation is active only when the connector sets +`request_cache_events_enabled = True`, and calls its optional method: + +```python +record_request_cache_event(request, event, **fields) +``` + +SparkCache may enable that property with `SPARK_CONTEXT_CACHE_TRACE_REUSE=1`. +Connectors without the property receive no callbacks or dispatch metadata. +The runtime does not add Prometheus labels or public API fields. The connector +owns bounded request ledgers, aggregate counters, and opt-in request traces. + +Each event includes `preemptions`, copied from `request.num_preemptions`. +Token counts refer to the original prompt and exclude generated tokens. + +| Event | Fields | Authoritative boundary | +|---|---|---| +| `admitted` | `local_tokens`, `external_tokens`, `lease_attached`, `source` | Successful block allocation after local/remote reconciliation; `source` is `gpu_lease` or `prefix_lookup` | +| `restore_finalized` | `success`, `valid_prefix_tokens` | All-worker receive finalization after failure handling and the full-hit sampling-token adjustment | +| `prompt_step_completed` | `start_token`, `end_token`, `stale=False` | Model output that passes invalid-load, abort, stale-output, and attempt-generation checks | +| `preempted` | Common fields only | Request counters reset and preemption generation incremented | +| `finished` | `status` as the request-status enum name | Terminal request cleanup, before the connector's finish hook | + +The admission event describes reuse accepted for allocation, not completed +inference. External reuse remains provisional until receive finalization. +The connector must reconcile a failed restore against `valid_prefix_tokens`; +a failure does not increment the preemption generation. A replacement lookup +can therefore admit a prefix in the same generation. An ordinary resume after +an asynchronous load does not emit another admission event. Lease attribution +survives deferred block allocation and is emitted once allocation succeeds; +its saved generation cannot be reused after preemption. + +A full external prompt restore can materialize every prompt token while the +scheduler recomputes its final token to produce sampling logits. Restored state +span and external prompt tokens reused are separate quantities. A resident GPU +lease is local reuse, even when another request originally populated that +lease through persistent restoration. + +## Completed prompt work + +The scheduler snapshots each dispatched token range before advancing request +counters. Each range is clipped to the original prompt length and retains its +preemption generation. Matching accepted output emits that saved range once; +subsequent scheduling or counter resets cannot change its endpoints. + +A completed decode step can emit an empty prompt range. This allows the +connector to commit pending prefix reuse after a resumed request completes +inference without additional prompt computation. It contributes zero prompt +compute tokens. Stale, failed, or aborted output earns no completed-work +credit. These counters do not measure discarded GPU execution or kernel time. + +Cumulative prompt work and reuse across preemption attempts can exceed the +original prompt length. Do not derive actual computation by subtracting cache +offers from the prompt length. Request completion and missing-event handling +remain connector responsibilities. + +## Offline validation + +```bash +python -m pytest runtime/glm53-spark-mtp3-mesh/performance/attribution -q +``` + +Tests execute transformed scheduler methods and admission/output branches. +They cover partial local tails, lease adoption, prompt clipping, asynchronous +counter advancement, duplicate completion, failed restores, stale output, +preemption, terminal cleanup, and connectors without instrumentation. They do +not qualify CUDA execution, cache restoration, or hardware serving behavior. diff --git a/runtime/glm53-spark-mtp3-mesh/performance/attribution/patch_scheduler.py b/runtime/glm53-spark-mtp3-mesh/performance/attribution/patch_scheduler.py new file mode 100644 index 00000000..43a9abba --- /dev/null +++ b/runtime/glm53-spark-mtp3-mesh/performance/attribution/patch_scheduler.py @@ -0,0 +1,127 @@ +"""Instrument source-pinned MTP3 scheduler decisions without changing cache policy.""" + +import ast +import hashlib +from pathlib import Path + +BEFORE_SHA256 = "ce9460834e08f97dbbfeb3f1238b78ee6a3363dd59a2aef8ceeb715857385895" +AFTER_SHA256 = "c32704ef04ef8d04e5d785620437b4a8c01fdf7a2aeb148116b5aa2c259b5d76" + +METHODS = ''' def _sparkcache_record_event(self, request, event, **fields): + connector = self.connector + if not getattr(connector, "request_cache_events_enabled", False): + return + callback = getattr(connector, "record_request_cache_event", None) + if callback is not None: + fields.setdefault("preemptions", request.num_preemptions) + callback(request, event, **fields) + + def _sparkcache_capture_prompt_steps(self, scheduler_output): + if not getattr(self.connector, "request_cache_events_enabled", False): + return + # Preserve dispatch positions: async scheduling advances request counters + # before the matching output arrives, and preemption can reset them. + steps = {} + for req_id, count in scheduler_output.num_scheduled_tokens.items(): + request = self.requests[req_id] + start = min(request.num_computed_tokens, request.num_prompt_tokens) + end = min(request.num_computed_tokens + count, request.num_prompt_tokens) + steps[req_id] = (start, end, request.num_preemptions) + scheduler_output._sparkcache_prompt_steps = steps + + def _sparkcache_complete_prompt_step(self, scheduler_output, request, stale): + step = getattr(scheduler_output, "_sparkcache_prompt_steps", {}).pop( + request.request_id, None + ) + if step is None or stale or step[2] != request.num_preemptions: + return + self._sparkcache_record_event( + request, "prompt_step_completed", start_token=step[0], + end_token=step[1], preemptions=step[2], stale=False, + ) + +''' + +TRANSFORMS = ( + (' def _preempt_request(\n', METHODS + ' def _preempt_request(\n'), + (' local_lease_alternative = None\n', + ''' local_lease_alternative = None + cache_trace_lease_attached = ( + getattr(request, "_sparkcache_pending_lease_generation", None) + == request.num_preemptions + ) +'''), + (' if attached_tokens:\n', + ''' if attached_tokens: + cache_trace_lease_attached = True + # Allocation can defer this attached request; retain + # attribution until admission in the same attempt. + request._sparkcache_pending_lease_generation = request.num_preemptions +'''), + (' # Record at admission so unscheduled lookups are not counted.\n', + ''' if did_prefix_cache_lookup or cache_trace_lease_attached: + cache_trace_local = min( + num_new_local_computed_tokens if did_prefix_cache_lookup + else request.num_computed_tokens, request.num_prompt_tokens + ) + self._sparkcache_record_event( + request, "admitted", local_tokens=cache_trace_local, + external_tokens=min(num_external_computed_tokens, + request.num_prompt_tokens - cache_trace_local), + lease_attached=cache_trace_lease_attached, + source="gpu_lease" if cache_trace_lease_attached else "prefix_lookup", + ) + request._sparkcache_pending_lease_generation = None + + # Record at admission so unscheduled lookups are not counted. +'''), + (' def _update_after_schedule(self, scheduler_output: SchedulerOutput) -> None:\n', + ' def _update_after_schedule(self, scheduler_output: SchedulerOutput) -> None:\n self._sparkcache_capture_prompt_steps(scheduler_output)\n'), + (' request.num_preemptions += 1\n', + ' request.num_preemptions += 1\n self._sparkcache_record_event(request, "preempted")\n'), + (' req_index = model_runner_output.req_id_to_index[req_id]\n', + ''' self._sparkcache_complete_prompt_step( + scheduler_output, request, output_is_stale + ) + req_index = model_runner_output.req_id_to_index[req_id] +'''), + (' if request.request_id in self.failed_recving_kv_req_ids:\n', + ''' cache_trace_restore_failed = request.request_id in self.failed_recving_kv_req_ids + if request.request_id in self.failed_recving_kv_req_ids: +'''), + (' self.finished_recving_kv_req_ids.remove(request.request_id)\n', + ''' self._sparkcache_record_event( + request, "restore_finalized", success=not cache_trace_restore_failed, + valid_prefix_tokens=min(request.num_computed_tokens, request.num_prompt_tokens), + ) + self.finished_recving_kv_req_ids.remove(request.request_id) +'''), + (' self._inflight_prefills.discard(request)\n connector_delay_free_blocks, kv_xfer_params = self._connector_finished(request)\n', + ''' self._sparkcache_record_event(request, "finished", status=request.status.name) + self._inflight_prefills.discard(request) + connector_delay_free_blocks, kv_xfer_params = self._connector_finished(request) +'''), +) + + +def transform(source: bytes) -> bytes: + if hashlib.sha256(source).hexdigest() != BEFORE_SHA256: + raise ValueError("Cache-attribution scheduler preimage differs") + text = source.decode().replace("\r\n", "\n") + for before, after in TRANSFORMS: + if text.count(before) != 1: + raise ValueError("Cache-attribution scheduler anchor differs") + text = text.replace(before, after, 1) + ast.parse(text) + return text.encode() + + +def apply(path: Path) -> dict: + source = path.read_bytes() + if hashlib.sha256(source).hexdigest() == AFTER_SHA256: + return {"before_sha256": BEFORE_SHA256, "after_sha256": AFTER_SHA256} + patched = transform(source) + if hashlib.sha256(patched).hexdigest() != AFTER_SHA256: + raise ValueError("Cache-attribution scheduler postimage differs") + path.write_bytes(patched) + return {"before_sha256": BEFORE_SHA256, "after_sha256": AFTER_SHA256} diff --git a/runtime/glm53-spark-mtp3-mesh/performance/attribution/test_connector_integration.py b/runtime/glm53-spark-mtp3-mesh/performance/attribution/test_connector_integration.py new file mode 100644 index 00000000..1558e4ab --- /dev/null +++ b/runtime/glm53-spark-mtp3-mesh/performance/attribution/test_connector_integration.py @@ -0,0 +1,186 @@ +"""Execute scheduler observations against a companion SparkCache token ledger. + +Set SPARKCACHE_SOURCE to a SparkCache checkout containing request_attribution.py. +The companion check is optional; an explicitly selected invalid checkout fails. +""" + +import ast +import importlib.util +import os +from pathlib import Path +import sys +from types import SimpleNamespace + +import pytest + +from test_patch_scheduler import execute_statements, process_output, request, runtime + + +@pytest.fixture +def ledger_type(): + checkout = os.environ.get("SPARKCACHE_SOURCE") + if not checkout: + pytest.skip("SPARKCACHE_SOURCE selects the companion token ledger") + source = Path(checkout) / "sparkcache/request_attribution.py" + if not source.is_file(): + pytest.fail(f"Companion token ledger is missing: {source}") + name = "sparkcache_companion_request_attribution" + spec = importlib.util.spec_from_file_location(name, source) + module = importlib.util.module_from_spec(spec) + sys.modules[name] = module + spec.loader.exec_module(module) + return module.RequestAttribution + + +def coupled(ledger_type, **overrides): + req = request(**overrides) + ledger = ledger_type(req.num_prompt_tokens) + cls, methods = runtime() + obj = cls() + obj.requests = {req.request_id: req} + summaries = [] + + def record(req, event, **fields): + if event == "finished": + summaries.append(ledger.summary(fields["status"])) + else: + ledger.record(event, **fields) + + obj.connector = SimpleNamespace(request_cache_events_enabled=True, + record_request_cache_event=record) + return obj, req, ledger, methods, summaries + + +def admit(obj, req, methods, local, external=0): + statement = next(node for node in ast.walk(methods["schedule"]) + if isinstance(node, ast.If) + and ast.unparse(node.test) == "did_prefix_cache_lookup or cache_trace_lease_attached") + execute_statements([statement], dict(self=obj, request=req, + did_prefix_cache_lookup=True, cache_trace_lease_attached=False, + num_new_local_computed_tokens=local, num_external_computed_tokens=external)) + + +def restore(obj, req, prefix, failed=False): + req.num_computed_tokens = prefix + obj.failed_recving_kv_req_ids = {req.request_id} if failed else set() + obj.finished_recving_kv_req_ids = {req.request_id} + obj.kv_cache_manager = SimpleNamespace(cache_blocks=lambda *args: None, + free=lambda *args: None) + obj.needs_kv_cache_zeroing = False + obj._update_waiting_for_remote_kv(req) + + +def dispatch(obj, req, start, count): + req.num_computed_tokens = start + output = SimpleNamespace(num_scheduled_tokens={req.request_id: count}) + obj._sparkcache_capture_prompt_steps(output) + return output + + +def finish(obj, req, summaries, status="FINISHED_STOPPED"): + req.status = SimpleNamespace(name=status) + req.is_finished = lambda: True + obj._inflight_prefills = SimpleNamespace(discard=lambda req: None) + obj._connector_finished = lambda req: (False, None) + obj.ec_connector = None + obj.encoder_cache_manager = SimpleNamespace(free=lambda req: None) + obj.finished_req_ids = set() + obj.finished_req_ids_dict = None + obj._free_blocks = lambda req: None + obj._free_request(req) + return summaries[-1] + + +def test_verified_external_prefix_and_async_output_account_once(ledger_type): + obj, req, ledger, methods, summaries = coupled(ledger_type) + admit(obj, req, methods, 256, 744) + restore(obj, req, 1000) + assert req.num_computed_tokens == 999 + output = dispatch(obj, req, 999, 32) + req.num_computed_tokens = 1100 + process_output(obj, methods, output, req) + process_output(obj, methods, output, req) + result = finish(obj, req, summaries) + assert result["attribution_complete"] + assert (result["local_tokens_reused"], result["external_tokens_reused"], + result["prompt_tokens_computed"]) == (256, 743, 1) + + +def test_failed_restore_and_same_generation_local_readmission(ledger_type): + obj, req, ledger, methods, summaries = coupled(ledger_type) + admit(obj, req, methods, 0, 1000) + restore(obj, req, 0, failed=True) + admit(obj, req, methods, 512) + output = dispatch(obj, req, 512, 488) + process_output(obj, methods, output, req) + result = finish(obj, req, summaries) + assert result["attribution_complete"] + assert (result["local_tokens_reused"], result["external_tokens_reused"], + result["prompt_tokens_computed"]) == (512, 0, 488) + + +def test_preemption_preserves_accepted_work_and_discards_inflight_output(ledger_type): + obj, req, ledger, methods, summaries = coupled( + ledger_type, status="running", spec_token_ids=[], num_output_placeholders=0) + admit(obj, req, methods, 256) + output = dispatch(obj, req, 256, 256) + process_output(obj, methods, output, req) + stale_output = dispatch(obj, req, 512, 32) + req.num_in_flight_tokens = 32 + obj._free_request_blocks = lambda req: None + obj.encoder_cache_manager = SimpleNamespace(free=lambda req: None) + obj._inflight_prefills = SimpleNamespace(discard=lambda req: None) + obj.log_stats = False + obj.waiting = SimpleNamespace(prepend_request=lambda req: None) + obj.reset_preempted_req_ids = set() + obj._preempt_request(req, 1.0) + process_output(obj, methods, stale_output, req) + admit(obj, req, methods, 0) + output = dispatch(obj, req, 0, 1000) + process_output(obj, methods, output, req) + result = finish(obj, req, summaries) + assert result["attribution_complete"] + assert result["preemptions"] == 1 + assert (result["local_tokens_reused"], result["external_tokens_reused"], + result["prompt_tokens_computed"]) == (256, 0, 1256) + + +def test_abort_does_not_credit_a_materialized_external_prefix(ledger_type): + obj, req, ledger, methods, summaries = coupled(ledger_type) + admit(obj, req, methods, 0, 1000) + restore(obj, req, 1000) + output = dispatch(obj, req, 999, 32) + req.is_finished = lambda: True + process_output(obj, methods, output, req) + result = finish(obj, req, summaries, "FINISHED_ABORTED") + assert not result["attribution_complete"] + assert result["external_tokens_reused"] == result["prompt_tokens_computed"] == 0 + + +def test_deferred_gpu_lease_admission_is_credited_only_after_execution(ledger_type): + obj, req, ledger, methods, summaries = coupled( + ledger_type, num_computed_tokens=0, prefill_stats=None, has_encoder_inputs=False) + nodes = list(ast.walk(methods["schedule"])) + initialize = next(node for node in nodes if isinstance(node, ast.Assign) + and ast.unparse(node.targets[0]) == "cache_trace_lease_attached") + attach = next(node for node in nodes if isinstance(node, ast.If) + and ast.unparse(node.test) == "attached_tokens") + allocate = next(node for node in nodes if isinstance(node, ast.If) + and ast.unparse(node.test) == "new_blocks is None") + admission = next(node for node in nodes if isinstance(node, ast.If) + and ast.unparse(node.test) == "did_prefix_cache_lookup or cache_trace_lease_attached") + values = dict(self=obj, request=req, request_id=req.request_id, lease_key="lease", + attached_tokens=768, did_prefix_cache_lookup=False, + num_new_local_computed_tokens=0, num_external_computed_tokens=0, + new_blocks=None) + execute_statements([initialize, attach, allocate, admission], values) + assert ledger.attempt is None + values["new_blocks"] = object() + execute_statements([initialize, allocate, admission], values) + assert ledger.local_tokens_reused == ledger.external_tokens_reused == 0 + output = dispatch(obj, req, req.num_computed_tokens, 232) + process_output(obj, methods, output, req) + result = finish(obj, req, summaries) + assert result["attribution_complete"] + assert (result["local_tokens_reused"], result["external_tokens_reused"], + result["prompt_tokens_computed"]) == (768, 0, 232) diff --git a/runtime/glm53-spark-mtp3-mesh/performance/attribution/test_patch_scheduler.py b/runtime/glm53-spark-mtp3-mesh/performance/attribution/test_patch_scheduler.py new file mode 100644 index 00000000..42c45842 --- /dev/null +++ b/runtime/glm53-spark-mtp3-mesh/performance/attribution/test_patch_scheduler.py @@ -0,0 +1,223 @@ +"""Execute attribution seams extracted from the transformed scheduler source.""" + +import ast +import hashlib +import importlib.util +from pathlib import Path +from types import SimpleNamespace + +import pytest + +HERE = Path(__file__).resolve().parent +SPEC = importlib.util.spec_from_file_location("cache_attribution_patch", HERE / "patch_scheduler.py") +PATCH = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(PATCH) +SOURCE = HERE.parent / "checkpoints/payload-by-sha" / PATCH.BEFORE_SHA256 / "scheduler.py" + + +def runtime(): + tree = ast.parse(PATCH.transform(SOURCE.read_bytes())) + cls = next(node for node in tree.body if isinstance(node, ast.ClassDef) and node.name == "Scheduler") + methods = {node.name: node for node in cls.body if isinstance(node, ast.FunctionDef)} + chosen = [node for name, node in methods.items() if name.startswith("_sparkcache_")] + chosen.extend(methods[name] for name in ("_update_waiting_for_remote_kv", "_preempt_request", "_free_request")) + module = ast.Module(body=[ast.ImportFrom(module="__future__", names=[ast.alias(name="annotations")], level=0), + ast.ClassDef(name="Harness", bases=[], keywords=[], body=chosen, decorator_list=[])], type_ignores=[]) + namespace = {"RequestStatus": SimpleNamespace(RUNNING="running", PREEMPTED="preempted")} + exec(compile(ast.fix_missing_locations(module), "scheduler-seams", "exec"), namespace) + return namespace["Harness"], methods + + +def execute_statements(statements, values): + # The scheduler's continue statements retain their real control flow. + body = [ast.For(target=ast.Name(id="_once", ctx=ast.Store()), + iter=ast.Tuple(elts=[ast.Constant(1)], ctx=ast.Load()), + body=statements, orelse=[])] + module = ast.Module(body=body, type_ignores=[]) + exec(compile(ast.fix_missing_locations(module), "scheduler-statements", "exec"), values) + + +def request(**overrides): + values = dict(request_id="request", num_computed_tokens=768, num_prompt_tokens=1000, + num_tokens=1000, num_preemptions=0, num_in_flight_tokens=32, + num_stale_output_tokens=0, drop_stale_output=False, + is_finished=lambda: False) + values.update(overrides) + return SimpleNamespace(**values) + + +def scheduler(req): + cls, methods = runtime() + events = [] + obj = cls() + obj.requests = {req.request_id: req} + obj.connector = SimpleNamespace(request_cache_events_enabled=True, + record_request_cache_event=lambda req, event, **fields: events.append((event, fields))) + return obj, events, methods + + +def test_transform_exact_source_and_idempotence(tmp_path): + target = tmp_path / "scheduler.py" + target.write_bytes(SOURCE.read_bytes()) + PATCH.apply(target) + assert hashlib.sha256(target.read_bytes()).hexdigest() == PATCH.AFTER_SHA256 + assert PATCH.apply(target)["after_sha256"] == PATCH.AFTER_SHA256 + target.write_bytes(target.read_bytes() + b"# unsupported\n") + with pytest.raises(ValueError, match="preimage"): + PATCH.apply(target) + + +@pytest.mark.parametrize("local,external,lease,lookup,expected", [ + (767, 0, False, True, 767), # Retained partial tail, not rounded connector argument. + (512, 488, False, True, 512), + (1024, 512, False, True, 1000), + (512, 0, True, False, 768), + (0, 0, False, False, None), # Resume after restore is not another admission. +]) +def test_admission_uses_adopted_prefix_not_lookup_offer(local, external, lease, lookup, expected): + req = request() + obj, events, methods = scheduler(req) + statement = next(node for node in ast.walk(methods["schedule"]) + if isinstance(node, ast.If) and ast.unparse(node.test) == "did_prefix_cache_lookup or cache_trace_lease_attached") + execute_statements([statement], dict(self=obj, request=req, + did_prefix_cache_lookup=lookup, cache_trace_lease_attached=lease, + num_new_local_computed_tokens=local, num_external_computed_tokens=external)) + if expected is None: + assert events == [] + else: + assert events == [("admitted", dict(local_tokens=expected, external_tokens=min(external,1000-expected), + lease_attached=lease, source="gpu_lease" if lease else "prefix_lookup", preemptions=0))] + + +def process_output(obj, methods, output, req, failed=False): + loop = next(node for node in ast.walk(methods["update_from_output"]) + if isinstance(node, ast.For) and ast.unparse(node.target) == "(req_id, num_tokens_scheduled)") + # Execute the real failure, abort and stale-output checks through the hook. + end = next(i for i, node in enumerate(loop.body) + if isinstance(node, ast.Expr) and isinstance(node.value, ast.Call) + and isinstance(node.value.func, ast.Attribute) + and node.value.func.attr == "_sparkcache_complete_prompt_step") + execute_statements(loop.body[:end + 1], dict(self=obj, scheduler_output=output, + req_id=req.request_id, num_tokens_scheduled=32, + failed_kv_load_req_ids={req.request_id} if failed else set())) + + +def test_completed_ranges_survive_async_counter_advance_and_exclude_decode(): + req = request(num_computed_tokens=992) + obj, events, methods = scheduler(req) + output = SimpleNamespace(num_scheduled_tokens={req.request_id: 32}) + obj._sparkcache_capture_prompt_steps(output) + req.num_computed_tokens = 1100 + process_output(obj, methods, output, req) + assert events == [("prompt_step_completed", dict(start_token=992, end_token=1000, + preemptions=0, stale=False))] + obj._sparkcache_complete_prompt_step(output, req, False) + assert len(events) == 1 + + +@pytest.mark.parametrize("failure", ["invalid_restore", "abort", "stale", "stale_drop", "preempted"]) +def test_failed_aborted_and_stale_output_cannot_credit_prompt_work(failure): + req = request() + obj, events, methods = scheduler(req) + output = SimpleNamespace(num_scheduled_tokens={req.request_id: 32}) + obj._sparkcache_capture_prompt_steps(output) + if failure == "abort": + req.is_finished = lambda: True + if failure in ("stale", "stale_drop"): + req.num_stale_output_tokens = 32 + req.drop_stale_output = failure == "stale_drop" + if failure == "preempted": + req.num_preemptions = 1 + process_output(obj, methods, output, req, failed=failure == "invalid_restore") + assert events == [] + + +def test_first_decode_output_can_commit_admitted_reuse_without_prompt_compute(): + req = request(num_computed_tokens=1024, num_preemptions=1) + obj, events, methods = scheduler(req) + output = SimpleNamespace(num_scheduled_tokens={req.request_id: 32}) + obj._sparkcache_capture_prompt_steps(output) + process_output(obj, methods, output, req) + assert events[0][1] == dict(start_token=1000, end_token=1000, preemptions=1, stale=False) + + +@pytest.mark.parametrize("failed,valid,expected", [(False, 1000, 999), (True, 0, 0), (True, 512, 512)]) +def test_restore_finalization_reports_effective_prefix_after_clamp_or_failure(failed, valid, expected): + req = request(num_computed_tokens=valid) + obj, events, _ = scheduler(req) + obj.failed_recving_kv_req_ids = {req.request_id} if failed else set() + obj.finished_recving_kv_req_ids = {req.request_id} + obj.kv_cache_manager = SimpleNamespace(cache_blocks=lambda *args: None, free=lambda *args: None) + obj.needs_kv_cache_zeroing = False + obj._update_waiting_for_remote_kv(req) + assert events == [("restore_finalized", dict(success=not failed, + valid_prefix_tokens=expected, preemptions=0))] + assert not obj.finished_recving_kv_req_ids + + +def test_unknown_or_disabled_connector_needs_no_accounting_metadata(): + req = request() + obj, events, _ = scheduler(req) + obj.connector = SimpleNamespace() + output = SimpleNamespace(num_scheduled_tokens={req.request_id: 32}) + obj._sparkcache_capture_prompt_steps(output) + obj._sparkcache_record_event(req, "admitted") + assert not hasattr(output, "_sparkcache_prompt_steps") + assert events == [] + + +def test_preemption_and_terminal_hooks_keep_attempt_generation_and_status(): + req = request(status="running", spec_token_ids=[], num_output_placeholders=0) + obj, events, _ = scheduler(req) + obj._free_request_blocks = lambda req: None + obj.encoder_cache_manager = SimpleNamespace(free=lambda req: None) + obj._inflight_prefills = SimpleNamespace(discard=lambda req: None) + obj.log_stats = False + obj.waiting = SimpleNamespace(prepend_request=lambda req: None) + obj.reset_preempted_req_ids = set() + obj._preempt_request(req, 1.0) + assert req.num_computed_tokens == 0 + assert events == [("preempted", {"preemptions": 1})] + req.status = SimpleNamespace(name="FINISHED_ABORTED") + req.is_finished = lambda: True + obj._connector_finished = lambda req: (False, None) + obj.ec_connector = None + obj.finished_req_ids = set() + obj.finished_req_ids_dict = None + obj._free_blocks = lambda req: None + obj._free_request(req) + assert events[-1] == ("finished", {"status": "FINISHED_ABORTED", "preemptions": 1}) + + +def test_lease_attribution_survives_allocation_deferral_until_accepted_output(): + req = request(num_computed_tokens=0, prefill_stats=None, has_encoder_inputs=False) + obj, events, methods = scheduler(req) + schedule_nodes = list(ast.walk(methods["schedule"])) + initialize = next(node for node in schedule_nodes if isinstance(node, ast.Assign) + and ast.unparse(node.targets[0]) == "cache_trace_lease_attached") + attach = next(node for node in schedule_nodes if isinstance(node, ast.If) + and ast.unparse(node.test) == "attached_tokens") + allocate = next(node for node in schedule_nodes if isinstance(node, ast.If) + and ast.unparse(node.test) == "new_blocks is None") + admit = next(node for node in schedule_nodes if isinstance(node, ast.If) + and ast.unparse(node.test) == "did_prefix_cache_lookup or cache_trace_lease_attached") + values = dict(self=obj, request=req, request_id=req.request_id, lease_key="lease", + attached_tokens=768, did_prefix_cache_lookup=False, + num_new_local_computed_tokens=0, num_external_computed_tokens=0, + new_blocks=None) + execute_statements([initialize, attach, allocate, admit], values) + assert req.num_computed_tokens == 768 + assert req._sparkcache_pending_lease_generation == 0 + assert events == [] + values["new_blocks"] = object() + execute_statements([initialize, allocate, admit], values) + assert events[0] == ("admitted", dict(local_tokens=768, external_tokens=0, + lease_attached=True, source="gpu_lease", preemptions=0)) + assert req._sparkcache_pending_lease_generation is None + execute_statements([initialize, allocate, admit], values) + assert len(events) == 1 + output = SimpleNamespace(num_scheduled_tokens={req.request_id: 32}) + obj._sparkcache_capture_prompt_steps(output) + process_output(obj, methods, output, req) + assert events[-1] == ("prompt_step_completed", dict(start_token=768, end_token=800, + preemptions=0, stale=False)) diff --git a/runtime/glm53-spark-mtp3-mesh/performance/install.py b/runtime/glm53-spark-mtp3-mesh/performance/install.py index 3653fdfc..d0ef6b44 100644 --- a/runtime/glm53-spark-mtp3-mesh/performance/install.py +++ b/runtime/glm53-spark-mtp3-mesh/performance/install.py @@ -69,6 +69,16 @@ path = SITE / row["path"] if row["sha256"] != hashlib.sha256(path.read_bytes()).hexdigest(): raise ValueError(f"Ownership dependency differs: {row['path']}") +# Apply the named transform only after every checkpoint ownership preimage passed. +sys.path.insert(0, str(SOURCE / "attribution")) +from patch_scheduler import apply as apply_attribution, BEFORE_SHA256, AFTER_SHA256 + +attribution_transform = apply_attribution(scheduler) +for row in data["files"]: + if row["path"] == "vllm/v1/core/sched/scheduler.py": + if row["sha256"] != BEFORE_SHA256: + raise ValueError("Attribution ownership scheduler preimage differs") + row["sha256"] = AFTER_SHA256 contract.write_text(json.dumps(data, indent=2) + "\n", encoding="utf-8") shutil.copytree(SOURCE / "bundle", Path("/opt/spark-sircl"), dirs_exist_ok=True) shutil.copyfile( @@ -103,6 +113,7 @@ "schema": "sparkring-mtp3-performance-image/v1", "status": "research-only", "sparkcache_commit": context["sparkcache_commit"], + "runtime_transforms": {"request_cache_attribution": attribution_transform}, "files": files, }, sort_keys=True, diff --git a/runtime/glm53-spark-mtp3-mesh/performance/prepare.py b/runtime/glm53-spark-mtp3-mesh/performance/prepare.py index 519ccda0..8cecddfa 100644 --- a/runtime/glm53-spark-mtp3-mesh/performance/prepare.py +++ b/runtime/glm53-spark-mtp3-mesh/performance/prepare.py @@ -50,7 +50,7 @@ def prepare(cache, placement, transport, output): path = output / name path.parent.mkdir(parents=True, exist_ok=True) path.write_bytes(bundle.extractfile(member).read()) - for name in ("checkpoints", "reasoning"): + for name in ("checkpoints", "reasoning", "attribution"): shutil.copytree( HERE / name, output / name, From 9efba11925265543119573ef8b926dbfe3135999 Mon Sep 17 00:00:00 2001 From: FujitsuPolycom <87842395+FujitsuPolycom@users.noreply.github.com> Date: Sun, 6 Sep 2026 17:51:31 -0500 Subject: [PATCH 2/5] Preserve final-chunk continuation checkpoints in attributed image builds Install four byte-attested continuation sources after verifying checkpoint ownership, then apply the matching scheduler attribution transform. Preserve all unrelated dependencies and generate image receipts from the composed runtime. Enable both checkpoint flags and pin the SparkCache attribution source; published image receipts retain their identities. Avoid redundant empty prompt events during ordinary decode. Validation: 501 mesh tests passed, four optional skips, including both scheduler variants and companion-ledger cases; Ruff and 562 documentation links passed. Hardware validation remains pending. --- .../performance/Dockerfile | 4 +- .../performance/README.md | 12 +- .../performance/attribution/README.md | 8 +- .../attribution/patch_scheduler.py | 29 ++- .../attribution/test_connector_integration.py | 1 + .../attribution/test_patch_scheduler.py | 34 ++- .../performance/continuation/LICENSE | 201 ++++++++++++++++++ .../performance/continuation/NOTICE | 4 + .../performance/continuation/README.md | 64 ++++++ .../performance/continuation/install.py | 57 +++++ .../performance/continuation/manifest.json | 30 +++ .../performance/continuation/source.tar.gz | Bin 0 -> 70110 bytes .../performance/continuation/test_install.py | 67 ++++++ .../performance/install.py | 21 +- .../performance/prepare.py | 4 +- 15 files changed, 514 insertions(+), 22 deletions(-) create mode 100644 runtime/glm53-spark-mtp3-mesh/performance/continuation/LICENSE create mode 100644 runtime/glm53-spark-mtp3-mesh/performance/continuation/NOTICE create mode 100644 runtime/glm53-spark-mtp3-mesh/performance/continuation/README.md create mode 100644 runtime/glm53-spark-mtp3-mesh/performance/continuation/install.py create mode 100644 runtime/glm53-spark-mtp3-mesh/performance/continuation/manifest.json create mode 100644 runtime/glm53-spark-mtp3-mesh/performance/continuation/source.tar.gz create mode 100644 runtime/glm53-spark-mtp3-mesh/performance/continuation/test_install.py diff --git a/runtime/glm53-spark-mtp3-mesh/performance/Dockerfile b/runtime/glm53-spark-mtp3-mesh/performance/Dockerfile index d248198b..06fb8ff7 100644 --- a/runtime/glm53-spark-mtp3-mesh/performance/Dockerfile +++ b/runtime/glm53-spark-mtp3-mesh/performance/Dockerfile @@ -1,10 +1,10 @@ ARG PARENT_IMAGE=ghcr.io/fujitsupolycom/sparkring-glm53-sparkcache@sha256:67dc0ae453baaae6831ccec1d259b4ef8b236a8b0dc9f747d901b95c66ec1987 FROM ${PARENT_IMAGE} RUN --mount=type=bind,target=/performance python3 -S -B /performance/install.py -ENV SPARK_GDN_PREFILL_CHECKPOINTS=2 +ENV SPARK_GDN_PREFILL_CHECKPOINTS=2 SPARK_GDN_CONTINUATION_CHECKPOINTS=1 LABEL org.sparkring.runtime.status="research-only" \ org.sparkring.runtime.profile="glm53-mtp3-cache-checkpoints" \ - org.sparkcache.commit="48bbd2be4a7b972e56632a2d7b934bac5460f272" \ + org.sparkcache.commit="19873f697c1ebdaf2b11d2013f2411b31f9e0f81" \ org.sparkcache.cuda-placement-sha256="2657cdd2e54a097c9544e4c79ae62c0646db6db123ff24e4f0c384238c3a1e8d" \ org.sparkring.sircl.manifest-sha256="c0fd5567442b08b908cc193f36d0864e262573c7e5d232509479a823cface742" ENTRYPOINT ["python3", "-S", "-B", "/opt/sparkring/bin/start-performance.py"] diff --git a/runtime/glm53-spark-mtp3-mesh/performance/README.md b/runtime/glm53-spark-mtp3-mesh/performance/README.md index 8ce99a8f..7973bb98 100644 --- a/runtime/glm53-spark-mtp3-mesh/performance/README.md +++ b/runtime/glm53-spark-mtp3-mesh/performance/README.md @@ -7,7 +7,7 @@ The recipe preserves the parent model weights and does not change host fabric. ## Build inputs -Use a clean SparkCache checkout at `48bbd2be4a7b972e56632a2d7b934bac5460f272`. +Use a clean SparkCache checkout at `19873f697c1ebdaf2b11d2013f2411b31f9e0f81`. It contains the merged restore/publication improvements, periodic-capture option, and backlog gauges. Periodic full capture defaults off; enabling it trades more writes for shorter history reconstruction. @@ -93,3 +93,13 @@ local reuse, finalized persistent restoration, and accepted prompt work across preemption attempts. They are inactive unless the connector enables request cache events. Source availability does not change a published image receipt; a rebuilt image must be validated before serving evidence is claimed. + +## Continuation checkpoint sources + +Source builds preserve the [four continuation checkpoint files](continuation/README.md) +from the source-attested continuation serving image. Both recurrent checkpoint +flags are enabled in the build recipe. The installer validates checkpoint +ownership, applies those four replacements, then applies the matching request +attribution transform and generates the full image inventory. This source build +composition requires separate serving validation; it does not change the +immutable published image contract. diff --git a/runtime/glm53-spark-mtp3-mesh/performance/attribution/README.md b/runtime/glm53-spark-mtp3-mesh/performance/attribution/README.md index 7951432a..01ddf3da 100644 --- a/runtime/glm53-spark-mtp3-mesh/performance/attribution/README.md +++ b/runtime/glm53-spark-mtp3-mesh/performance/attribution/README.md @@ -3,8 +3,8 @@ Status: **implemented** with CPU scheduler-seam coverage. The MTP3 profile remains **research-only**; rebuilt-image validation is required. -`patch_scheduler.py` applies a byte-checked transform after the recurrent -checkpoint payload has been installed. The payload files and their manifest +`patch_scheduler.py` applies a byte-checked transform to either the recurrent +checkpoint scheduler or its explicitly pinned final-chunk continuation variant. The payload files and their manifest are immutable inputs. The image installer first verifies every checkpoint ownership dependency, applies the transform, and changes only the scheduler entry to the transform's expected output hash. The image receipt records both @@ -60,7 +60,9 @@ subsequent scheduling or counter resets cannot change its endpoints. A completed decode step can emit an empty prompt range. This allows the connector to commit pending prefix reuse after a resumed request completes inference without additional prompt computation. It contributes zero prompt -compute tokens. Stale, failed, or aborted output earns no completed-work +compute tokens. After that generation has consumed its admitted reuse, ordinary +decode steps do not allocate empty prompt-range entries or emit redundant events. +Stale, failed, or aborted output earns no completed-work credit. These counters do not measure discarded GPU execution or kernel time. Cumulative prompt work and reuse across preemption attempts can exceed the diff --git a/runtime/glm53-spark-mtp3-mesh/performance/attribution/patch_scheduler.py b/runtime/glm53-spark-mtp3-mesh/performance/attribution/patch_scheduler.py index 43a9abba..fbf552d8 100644 --- a/runtime/glm53-spark-mtp3-mesh/performance/attribution/patch_scheduler.py +++ b/runtime/glm53-spark-mtp3-mesh/performance/attribution/patch_scheduler.py @@ -5,7 +5,14 @@ from pathlib import Path BEFORE_SHA256 = "ce9460834e08f97dbbfeb3f1238b78ee6a3363dd59a2aef8ceeb715857385895" -AFTER_SHA256 = "c32704ef04ef8d04e5d785620437b4a8c01fdf7a2aeb148116b5aa2c259b5d76" +AFTER_SHA256 = "9500c99fd5f7d82e4c5247c41b64fd4cfc085ca39303db8991f4dcb7e5196a68" +CONTINUATION_BEFORE_SHA256 = "f46c40c1c41daf2bab4566dd185d320f4ec1fb11e8d632c90c47b0f1bb1808fa" +CONTINUATION_AFTER_SHA256 = "6c784cb6d30d89a078e386081f650e7249269704f2bea06c063c656c71bd9cf2" +SOURCE_TRANSFORMS = { + BEFORE_SHA256: AFTER_SHA256, + CONTINUATION_BEFORE_SHA256: CONTINUATION_AFTER_SHA256, +} + METHODS = ''' def _sparkcache_record_event(self, request, event, **fields): connector = self.connector @@ -26,7 +33,8 @@ def _sparkcache_capture_prompt_steps(self, scheduler_output): request = self.requests[req_id] start = min(request.num_computed_tokens, request.num_prompt_tokens) end = min(request.num_computed_tokens + count, request.num_prompt_tokens) - steps[req_id] = (start, end, request.num_preemptions) + if end > start or getattr(request, "_sparkcache_consumed_generation", None) != request.num_preemptions: + steps[req_id] = (start, end, request.num_preemptions) scheduler_output._sparkcache_prompt_steps = steps def _sparkcache_complete_prompt_step(self, scheduler_output, request, stale): @@ -35,10 +43,13 @@ def _sparkcache_complete_prompt_step(self, scheduler_output, request, stale): ) if step is None or stale or step[2] != request.num_preemptions: return + if step[0] == step[1] and getattr(request, "_sparkcache_consumed_generation", None) == step[2]: + return self._sparkcache_record_event( request, "prompt_step_completed", start_token=step[0], end_token=step[1], preemptions=step[2], stale=False, ) + request._sparkcache_consumed_generation = step[2] ''' @@ -72,6 +83,7 @@ def _sparkcache_complete_prompt_step(self, scheduler_output, request, stale): source="gpu_lease" if cache_trace_lease_attached else "prefix_lookup", ) request._sparkcache_pending_lease_generation = None + request._sparkcache_consumed_generation = None # Record at admission so unscheduled lookups are not counted. '''), @@ -105,7 +117,7 @@ def _sparkcache_complete_prompt_step(self, scheduler_output, request, stale): def transform(source: bytes) -> bytes: - if hashlib.sha256(source).hexdigest() != BEFORE_SHA256: + if hashlib.sha256(source).hexdigest() not in SOURCE_TRANSFORMS: raise ValueError("Cache-attribution scheduler preimage differs") text = source.decode().replace("\r\n", "\n") for before, after in TRANSFORMS: @@ -118,10 +130,13 @@ def transform(source: bytes) -> bytes: def apply(path: Path) -> dict: source = path.read_bytes() - if hashlib.sha256(source).hexdigest() == AFTER_SHA256: - return {"before_sha256": BEFORE_SHA256, "after_sha256": AFTER_SHA256} + before = hashlib.sha256(source).hexdigest() + for original, patched in SOURCE_TRANSFORMS.items(): + if before == patched: + return {"before_sha256": original, "after_sha256": patched} patched = transform(source) - if hashlib.sha256(patched).hexdigest() != AFTER_SHA256: + after = SOURCE_TRANSFORMS[before] + if hashlib.sha256(patched).hexdigest() != after: raise ValueError("Cache-attribution scheduler postimage differs") path.write_bytes(patched) - return {"before_sha256": BEFORE_SHA256, "after_sha256": AFTER_SHA256} + return {"before_sha256": before, "after_sha256": after} diff --git a/runtime/glm53-spark-mtp3-mesh/performance/attribution/test_connector_integration.py b/runtime/glm53-spark-mtp3-mesh/performance/attribution/test_connector_integration.py index 1558e4ab..b56642c4 100644 --- a/runtime/glm53-spark-mtp3-mesh/performance/attribution/test_connector_integration.py +++ b/runtime/glm53-spark-mtp3-mesh/performance/attribution/test_connector_integration.py @@ -14,6 +14,7 @@ import pytest from test_patch_scheduler import execute_statements, process_output, request, runtime +from test_patch_scheduler import scheduler_source # noqa: F401 @pytest.fixture diff --git a/runtime/glm53-spark-mtp3-mesh/performance/attribution/test_patch_scheduler.py b/runtime/glm53-spark-mtp3-mesh/performance/attribution/test_patch_scheduler.py index 42c45842..821b6476 100644 --- a/runtime/glm53-spark-mtp3-mesh/performance/attribution/test_patch_scheduler.py +++ b/runtime/glm53-spark-mtp3-mesh/performance/attribution/test_patch_scheduler.py @@ -5,6 +5,7 @@ import importlib.util from pathlib import Path from types import SimpleNamespace +import tarfile import pytest @@ -15,6 +16,16 @@ SOURCE = HERE.parent / "checkpoints/payload-by-sha" / PATCH.BEFORE_SHA256 / "scheduler.py" +@pytest.fixture(autouse=True, params=["fresh_prompt", "continuation"]) +def scheduler_source(request, monkeypatch, tmp_path): + if request.param == "continuation": + with tarfile.open(HERE.parent / "continuation/source.tar.gz") as archive: + data = archive.extractfile("vllm/v1/core/sched/scheduler.py").read() + source = tmp_path / "continuation-scheduler.py" + source.write_bytes(data) + monkeypatch.setitem(globals(), "SOURCE", source) + + def runtime(): tree = ast.parse(PATCH.transform(SOURCE.read_bytes())) cls = next(node for node in tree.body if isinstance(node, ast.ClassDef) and node.name == "Scheduler") @@ -59,9 +70,10 @@ def scheduler(req): def test_transform_exact_source_and_idempotence(tmp_path): target = tmp_path / "scheduler.py" target.write_bytes(SOURCE.read_bytes()) + expected = PATCH.SOURCE_TRANSFORMS[hashlib.sha256(SOURCE.read_bytes()).hexdigest()] PATCH.apply(target) - assert hashlib.sha256(target.read_bytes()).hexdigest() == PATCH.AFTER_SHA256 - assert PATCH.apply(target)["after_sha256"] == PATCH.AFTER_SHA256 + assert hashlib.sha256(target.read_bytes()).hexdigest() == expected + assert PATCH.apply(target)["after_sha256"] == expected target.write_bytes(target.read_bytes() + b"# unsupported\n") with pytest.raises(ValueError, match="preimage"): PATCH.apply(target) @@ -221,3 +233,21 @@ def test_lease_attribution_survives_allocation_deferral_until_accepted_output(): process_output(obj, methods, output, req) assert events[-1] == ("prompt_step_completed", dict(start_token=768, end_token=800, preemptions=0, stale=False)) + + +def test_ordinary_decode_avoids_repeated_empty_prompt_events(): + req = request(num_computed_tokens=1000) + obj, events, methods = scheduler(req) + first = SimpleNamespace(num_scheduled_tokens={req.request_id: 32}) + queued = SimpleNamespace(num_scheduled_tokens={req.request_id: 32}) + obj._sparkcache_capture_prompt_steps(first) + obj._sparkcache_capture_prompt_steps(queued) + process_output(obj, methods, first, req) + obj._sparkcache_complete_prompt_step(queued, req, False) + assert len(events) == 1 + following = SimpleNamespace(num_scheduled_tokens={req.request_id: 32}) + obj._sparkcache_capture_prompt_steps(following) + assert following._sparkcache_prompt_steps == {} + req.num_preemptions = 1 + obj._sparkcache_capture_prompt_steps(following) + assert following._sparkcache_prompt_steps[req.request_id] == (1000, 1000, 1) diff --git a/runtime/glm53-spark-mtp3-mesh/performance/continuation/LICENSE b/runtime/glm53-spark-mtp3-mesh/performance/continuation/LICENSE new file mode 100644 index 00000000..261eeb9e --- /dev/null +++ b/runtime/glm53-spark-mtp3-mesh/performance/continuation/LICENSE @@ -0,0 +1,201 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/runtime/glm53-spark-mtp3-mesh/performance/continuation/NOTICE b/runtime/glm53-spark-mtp3-mesh/performance/continuation/NOTICE new file mode 100644 index 00000000..5e7e22b1 --- /dev/null +++ b/runtime/glm53-spark-mtp3-mesh/performance/continuation/NOTICE @@ -0,0 +1,4 @@ +The archived source files derive from vLLM and preserve their Apache-2.0 +SPDX notices and vLLM contributor copyright statements. SparkRing changes +implement final-chunk recurrent checkpoint planning and allocation. +The manifest identifies exact source preimages and replacements. diff --git a/runtime/glm53-spark-mtp3-mesh/performance/continuation/README.md b/runtime/glm53-spark-mtp3-mesh/performance/continuation/README.md new file mode 100644 index 00000000..51fa27b1 --- /dev/null +++ b/runtime/glm53-spark-mtp3-mesh/performance/continuation/README.md @@ -0,0 +1,64 @@ +# Final-chunk recurrent checkpoints + +Status: **research-only**. The source package preserves the four continuation +files used by the serving image identified below. CPU tests verify package +integrity and ownership updates; they do not qualify the combined rebuilt image. + +`source.tar.gz` contains exact vLLM source bytes for: + +- `vllm/v1/core/kv_cache_manager.py` +- `vllm/v1/core/recurrent_prefill_checkpoint.py` +- `vllm/v1/core/sched/scheduler.py` +- `vllm/v1/core/single_type_kv_cache_manager.py` + +The [manifest](manifest.json) binds the archive hash, each source preimage and +replacement, and the originating overlay manifest. The archived files preserve +their upstream notices; [LICENSE](LICENSE) contains the Apache 2.0 license and +[NOTICE](NOTICE) identifies the local modifications. Compression prevents text +checkout conversion and formatters from changing source bytes. + +## Behavior + +With `SPARK_GDN_PREFILL_CHECKPOINTS=2` and +`SPARK_GDN_CONTINUATION_CHECKPOINTS=1`, the scheduler can retain explicit +recurrent checkpoints inside the final continuation chunk of a cold text +prefill. The chunk is capped at 8,192 tokens. Eligibility starts only after +successful uncached admission and is removed on cleanup or invalidation. +Preempted, resumed, encoder, speculative, and incompatible request states do +not use this path. + +The allocator validates the recurrent source and private speculative reserve, +preserves worker-visible block IDs, and allocates checkpoint destinations +without duplicate ownership. Publication boundaries must lie strictly inside +the selected chunk; an existing boundary at the chunk start is not a new export. + +## Provenance and composition + +The four files match the continuation serving image with config ID +`sha256:489d1975619e9083d14f14bcd1c6cbb4a96c41e3ab3978af048dc3ed2bb452a8`. +Its parent was the published cache/checkpoint image with config ID +`sha256:6921a6c163ea40b603e19a0332330efe3dbccbf4dce9f6cbbf6b756c9231835a`. +These identities establish source provenance, not qualification of another +image containing additional changes. + +The performance image installer validates all checkpoint ownership dependencies +before applying these four source replacements. `install.py` verifies the +manifest, archive, source preimages, and corresponding ownership entries before +writing any source file. It updates only those four ownership hashes and +preserves every symbol requirement and unrelated dependency. + +[Request-attribution instrumentation](../attribution/README.md) then applies its +separately pinned scheduler transform. The builder generates a complete installed +file inventory after both transforms. It does not copy the originating image's +performance or ownership receipts over the combined installation. Published +image contracts retain their immutable identities. + +## Offline checks + +```bash +python -m pytest runtime/glm53-spark-mtp3-mesh/performance/continuation -q +``` + +The tests reject altered packages, unexpected runtime preimages, and ownership +drift before any source write. Attribution tests execute scheduler boundaries +against both the fresh-prompt and continuation source variants. diff --git a/runtime/glm53-spark-mtp3-mesh/performance/continuation/install.py b/runtime/glm53-spark-mtp3-mesh/performance/continuation/install.py new file mode 100644 index 00000000..1d4f525a --- /dev/null +++ b/runtime/glm53-spark-mtp3-mesh/performance/continuation/install.py @@ -0,0 +1,57 @@ +"""Install four exact continuation sources after checkpoint ownership verification.""" + +import hashlib +import io +import json +from pathlib import Path +import tarfile + +HERE = Path(__file__).resolve().parent +MANIFEST_SHA256 = "ba458f84d1c07f5e070620629afc1259ca206feffb23ea9bf4f00931f79fbe7a" + + +def package(context=HERE): + raw = (context / "manifest.json").read_bytes() + if hashlib.sha256(raw).hexdigest() != MANIFEST_SHA256: + raise ValueError("Continuation source manifest differs") + manifest = json.loads(raw) + compressed = (context / "source.tar.gz").read_bytes() + if hashlib.sha256(compressed).hexdigest() != manifest["source_archive_sha256"]: + raise ValueError("Continuation source archive differs") + sources = {} + with tarfile.open(fileobj=io.BytesIO(compressed), mode="r:gz") as archive: + for member in archive.getmembers(): + if not member.isfile() or member.name not in manifest["files"] or member.name in sources: + raise ValueError("Continuation archive member differs") + content = archive.extractfile(member).read() + if hashlib.sha256(content).hexdigest() != manifest["files"][member.name]["after_sha256"]: + raise ValueError("Continuation source payload differs") + compile(content, member.name, "exec") + sources[member.name] = content + if sources.keys() != manifest["files"].keys(): + raise ValueError("Continuation source inventory differs") + return manifest, sources + + +def apply(site: Path, ownership: dict, context=HERE): + manifest, sources = package(context) + rows = {row["path"]: row for row in ownership["files"]} + if len(rows) != len(ownership["files"]): + raise ValueError("Duplicate ownership dependency") + writes = [] + for name, source in sources.items(): + expected = manifest["files"][name] + target = site / name + if not target.resolve().is_relative_to(site.resolve()): + raise ValueError("Continuation source target escapes installation") + if hashlib.sha256(target.read_bytes()).hexdigest() != expected["before_sha256"]: + raise ValueError(f"Continuation runtime preimage differs: {name}") + if name not in rows or rows[name]["sha256"] != expected["before_sha256"]: + raise ValueError(f"Continuation ownership preimage differs: {name}") + writes.append((target, source)) + # Validate all four source and ownership preimages before the first write. + for target, source in writes: + target.write_bytes(source) + for name, expected in manifest["files"].items(): + rows[name]["sha256"] = expected["after_sha256"] + return {"manifest_sha256": MANIFEST_SHA256, "files": manifest["files"]} diff --git a/runtime/glm53-spark-mtp3-mesh/performance/continuation/manifest.json b/runtime/glm53-spark-mtp3-mesh/performance/continuation/manifest.json new file mode 100644 index 00000000..629ef46e --- /dev/null +++ b/runtime/glm53-spark-mtp3-mesh/performance/continuation/manifest.json @@ -0,0 +1,30 @@ +{ + "environment": { + "SPARK_GDN_CONTINUATION_CHECKPOINTS": "1", + "SPARK_GDN_PREFILL_CHECKPOINTS": "2" + }, + "files": { + "vllm/v1/core/kv_cache_manager.py": { + "after_sha256": "26da2ce150958b9eb1495badfa5493f2164f8d86b943306b18b6b2abf83189c3", + "before_sha256": "f4ccf9da197eb68fd21b23e8c03615cd0d4c2f871da0dc32a9c6ec82e44d3931" + }, + "vllm/v1/core/recurrent_prefill_checkpoint.py": { + "after_sha256": "e29068047dbc505df14888f68f56745545873d3665c1db9640a53234c6a420dc", + "before_sha256": "51ef011d11bb374d61840f1ff819759b6e3dd325da41856b7b0f01b013b54713" + }, + "vllm/v1/core/sched/scheduler.py": { + "after_sha256": "f46c40c1c41daf2bab4566dd185d320f4ec1fb11e8d632c90c47b0f1bb1808fa", + "before_sha256": "ce9460834e08f97dbbfeb3f1238b78ee6a3363dd59a2aef8ceeb715857385895" + }, + "vllm/v1/core/single_type_kv_cache_manager.py": { + "after_sha256": "198dbd75397903d1e3378ac8af0ac0cd3f6b85fbb0f7bd5c65cd1780b2460a26", + "before_sha256": "10a6a60f31c67b59825aa827084f3d25416128ba5dc7c14a774675d64bbec40b" + } + }, + "parent_image_id": "sha256:6921a6c163ea40b603e19a0332330efe3dbccbf4dce9f6cbbf6b756c9231835a", + "schema": "sparkring-continuation-source/v1", + "source_archive_sha256": "91a411772951ef52f38e1d5224cd814e09ab9978d77a8fddcd665c9d5dfebf0c", + "source_overlay_manifest_sha256": "9e6b368508c30a4f443cb16da25230c7d982792aea1d99e90dd5f8f21e5c8429", + "source_role": "Final-chunk recurrent checkpoints for cold text prefills", + "status": "research-only" +} diff --git a/runtime/glm53-spark-mtp3-mesh/performance/continuation/source.tar.gz b/runtime/glm53-spark-mtp3-mesh/performance/continuation/source.tar.gz new file mode 100644 index 0000000000000000000000000000000000000000..c3f834e339efe1915b7bc79ed4f0de2b41c248ea GIT binary patch literal 70110 zcmV(;K-<3`iwFP!000023hce2kSBR~Pwmo0jQ$Ue9*dzxo3Ij^W=Qe)l{2FZ{Fr`|uC{`n!L9D8J+H zAO7z6(eHkh9RKR?@NeI=X${l=>i^DvcawK-p8d!4WwFZ3CZE2@^0F<~MPAR7lO5v1 z^wI42*LTEY-xQntX|=nqi}Q>2eg3(fC+cIes>-%5&h~9pH%VJ1?M0qkzI^#A+11t0 z`KtZ(onPN6w!5lslcLS*wyHLb`lcuL+*Y&P(CQI*XsJzFK>iFf&*_IbII zPi1MFt~P1YeD;eSe{Li?|yuWfO(Ejm?LbebGS3|m7*-#C4Xf;v~P<|Gux)^#ghJ% zkFT=g(tPgn;S8_jOkK%URn=KhrmzV4&gj>768QJW>$AGZ#Hya^$0q#qdEPEVqrlJO z&Kb8k0A!_BC?~pE z5qL?RH~USSRBMGo%v~nQRaJjV(uOF7KD@KMy~^{F$9f0PXZsDTFfB800H87#MV33? z-)nfZGTjhO;E~{Z?^o@<&J!jMFsB-PX6%6fJmZ}u?S8k(KSRx=Mgv@^DwhW&`sjXEm-UDb~qVo?NA6i(5Z0VC<|| z11R7^@sP`=_;Zdws_(<=DoR?_dXGmplgINM9!VOwHJcpCd~PEou#o*G!{yZZ4$uz< z&ocmlK-6in0b*~TCGBVn#0zFi%cLspA?1tkQ@-7`*NhkRlDKINfS|pA69~MQB%>6# zV7{uhJ7h4jl&3;^s*|lX!| z0A{O;v?xbsn@^r!_d;ORACa(KElWUebL?=Q2G_i#CJ zA^6|kbtHdHUenvga&9>P8Jy+d{mXeR{L>$Rw*p97CTU@8#M_}(sMe1E-V$Oq^8haT z$8aL9F_1a@4X!XCQ9kz(aB7fOeg)(lAS}D8%!s&u_hUj-D`Bb?8X-rLRg@(Rfs>KA z45-iBRGT5PFTW7fFKzwtlEmO19teT_>eC_S0v|`4yd2S6<7BZ&j_F#_XPm~kO=AZF z;$_VS=pNhQNp8k9b<< z)3|yZ9pU}4A3#euTEQ$GSSf)0Vh{th@Kdo%cEEs8)8f}>J}_*2`rxCp4Xqdl5a4q# z-$42p80iX>mDu*IXqM$(3PSSVNBd(8W%X0m*_8R!k|YCt2A%BIAa`}1wkVn57%_T> zvIT6whj2ev<{LU1(~g$7n>sDe^O1|@i;?B*d0lDt4biOz!v@$1lp|dV^LN6EK3$!oKl8#v6H=o^5hk zK3rP`9Ybiqf1o)m^AsfPAbXc5LWc(wXj4_6QWR2oYTz%%gP>L}NqdbBC6=Us%J!G& zrVpbaNDzEQvh-V9lCH9 zLclB5XOp=8bOFEb%O(fKeVw9$l>!BXt8{9;a+)-(UV?^kz1qN$PSY#6qac#(%;9jO z=?)EMLLJ@J1#E7#FX6#mo{f_<+ZK&b9YG<4UuUGvD=amKOE*{PH8a6YkvC!mpmSfQ z8&DAe8uh+JyWY$aOhV`?>9iww1RwR_Y>I`SvP8U&wsH3Yo^nKxcp7^M$aB|^a z^GQEuFG62B?W~2Nfa4*6yIM}`jf9b-ppL67VCmp+kgr_$&nHIVGCbrW)01&n~Jrvs(2fMV}!wxhG1*thg< zxBC{Us>}1}d2jKH0X9}&w~G;%@i<6}K#jks>UO$;JCOnTNl`}J-n#2ZGEt z;-d|oNn3r5-k&Tl6XoqH6S!^lwK)uX38v&F*hxuYxkg*4Jp7&A*3+4=1Eqp+7E!xlYoYdCfUFI&AtNPuh&yT1OV`rB$^0eM5xLI zJs$8btx=?cw+aj7)f$zL?CWHQ=0&(4AbD+sI8Jc6*U0fxa({v7=WR+1zQnye=_ufm%NH`6V?31zQ;p}+a zV8eBVT?OYL`3^{Z&BLN02V$AFfg( zMOSMmLsU(kSk^Gl>!GZgOr^W&_PPK^N7Hbjg_6hs!cXw9P(?> z-x(vU0mOS!mR|2S8;fL})aO3U5W9Gt{`D}pY^~&|S4C(>=#A4;TR=F2iot^KGgH)KrQTw@?$4O(piVmgq$EqN7VHrOCk z7m3d(5wiLNC0Cjw3`= zT{focK_jhqaXpo~m#2ic*rRGKfwC->mCX(I@Ki7_vN58FO~}sRsADe;^TO0JFCcl- zinw%fWkR&G41tz@#F-yb-oH=!*ff<{3u6`#S<<2KZs7e^$PFT+aSgKKGOy1O%d=dD zTcPC@P1Pr>7J%4w@zqBq{d;i^omC1OhgCG!idU zMZK8t1GQKo8sJM6#vr23mSU6zoV~g16cekA^HkToSUbQ`F&jlYk-;PEWMQ$~3^vi@ zot_465Nwfb{V@0n51ckxR$7XYDGF$L#sygz{JJWq7mURMSvgph!nV*uL>O5T#2RwB zL>4j7avI%2U7$%}WKbEQ@iUUiHyrniMS7+J6D8OM-#9%*sZ-poj8+;{TJ0bOR|fWU zx{%E5N*#bG%Pk_va+6+La-@{mibjSQj?C1sXinMKAHovU;jS*#U0q799=6v-w9T(* zDcQ=msWUr+Sx*A+Ye(r3uny@jIcpGfq(XS*mwNL3^8{0`;kGV>@=d zydEL7dE%_>gRxr<>c*!)_Oc&yp?eo?aVANl*92c6l83NLG>zZpDaM4kO2%E67YNDX z3Z*{5f4&yp;aroXEYZS%JshiB=r9!PO3&*YsA5&lz8q>I$eD;Ny1;oudCBtPzES+0 z(Y{HtifwHm|E^lJYD6U1k>VAW+bke^(N^d#Y>RT_9DMRvKVwQ>jtL|hQ|q}iY%rqS zXT-M#ehtM3JF%$NV=~RpcGJhgJ$Y{;tfy-Vcuw-aAxi?rmtI@v+Zl43bBb!mfJA#5 zk+ir(mcCpUZFErdN!FJJIQ$Wjl_IxW!0XkgR(Kyb-mo~v!iquSZ zvyyBtYSVkFsm#Lu7*#$OaYLt1Y!-pJsFIynGhuiR!)!g6VN-%gGd61J$I90~amld)VC#iN|7D&6GG3WPX=G8%Ok zl&5u@wq!uqVQ8XoF|IE5<)?syW+n04c1#r1nkE)szZ zQaTcPZe@bz%fZqH5{kihzJg1?E9d zPhcP7KB&EfuqNUX3f)ihh==N2VI@GW#L0c|xAJj9 zaGVUIWIjr1G(((0%Qq&{S&Ks*dgL$g#`qOFlbhk#DlLf`VWbAeIXHN;pT-7H(i2n$3~AEp$e6t}{+}ZEKSl0ah}`Rf zJSe~Lg_-TDU4XhwNQfNY_|EhVYb1{kybJ%dgZ6MeWZbbAZ^nY(z1`+Ntp9+1~uha)CQ53sD}K7y>|}SYjC>tjjaaB*^@= zNf^F)@QhvJQZZL$a`=&=y*qVgGR%fW!S>D|E3ygf~)VfKb$0dYKj*%47`U4y^<*X3B*P6i5l$5~aD$ zNV;nGK!FV-R?*P~JjTE;MiarIl7{C~bQ^UZ01sP6y-CQH$c)sY61fp`-%4>}r zNEGFSI;ISRb@lv9F0-S`uyfRB0>?w|NGOsk=80SFB3gsBWVH&{ zWC@2Qbjp3HYGq_~x)!Nnjae8^Qu;+ur;DBhXW3C1ZmndC8r?KK2tgot=~^8qjJnG) ztk`1dS5jOgv6ebG$T!3sqci-K6CPWwQDXe&PKEdj3g||`t!dNq+_X$g zlW*|*Jsv>bG$=9Xyq{A%w^OzypnPi~36~(JxOc;4WQvC)qn?5A^_dFGH2EWpyvcP* z69xdyiWw0SYws(>I(3~6@#S+KSD5;Z-9b*IdV_~;m>N#i#By~-iQ~Dq#U9?=Vr*o~ zY@-v%Pw}z)8f{O*)}ITACePIIWK{W;2BD}wxvm`fY%_J?kzH4jKB&isZ2#4x?&{osXjllPwS4H zxup7aPQjrfi1t_|7K^HUWtAKS#xn8?t=%1YIg$2uyMr_4efF{?VZkDa2X7dJdeaOa zFdnt7#&vVYq6z(!`qB9S#`L?1UWju}9=UB1P(iUiJ|vO&?NFf5{eJFfdM2wjI3dRE zhOCL9<=yY*lYtzBSk*&2)TCQo?7*x1^NyTDW)7Rbq4@UiyWBdSL>#RL5(upbOn<%0 zM?H=)xldKZ*zEHW+)+N)0TAPaP-W!@$z->1e8d56B0}FY zeIFv&azF9!h{ug7D=Jv9a7k4Hm#7booH@sVP=9+-PlCnsaG^M!%ux!`db5O_Eg*R? z#!{(+n0Rm;EO1Tn7g3SNpYVPXj{ouRbhKay>3c>87gcpimR8I5wbY%pKg}pV;}!du zNYPU&i&SsMRv2ea@gq$6&MM}E=u;Bfu<%q!@^2NggN5ae%47vw8k+OWDIJRbrWFVLX#4dt5Uf1E7Yy?Y|6E?&eEo!*bU`y zYV%!l&ww@q%@X5TA4J9}7f6;@u2i~f3%JB>u~K5eMDWa?IRqI!<#x#;Ixswix`J_w zma?g5@w((Qwv;XfmdGm<)(xUm9Z+<1Jc-TaNAYakd%v+Jtb(ulrvtyr;DJ6t-i9RakDxB1RxJACifJpAVT8-{iL z-oNaAgA#%}ktdu#6`|I?1+t^+Y-ee!h>1vO*#qoEp){qKCC|zN3#7br1i3U1xg|8) z%Blb;LxQMsGd9*V8L>cmDpSstI=D55sC8s8Rl(m)XXvb_wwel64G_RDp<~=yZTK<* z{LP}bJ1nBO2`5uO2GyJK@*zB{hX#kaQ=CY@dGKt`p>WEMspF}nGli?YT#{SZd>I88 zjN|!Xmr=B6D}u?mf{OGh1*AE%c860lIhaw{QM)5jEI}W_*soPNx~i&X|7ko&xmk9s z#YQ+Fv~j`QakAgxmjl@I-5%!b+aJka%%&>}MYZJNBkWeeYvIHb<%D9zz;r6IreSox zW!ZG>RK5hNfl~Q|)q$Pu!wb6ZCfwD4gOO}w)R8iu9hr50_@rk+wG7xFI~PM**^O+r zo%Y%0Y}SXnpDiS*`a+y;PO5Qql>(v=!+MlCR@ zgCwnv%G`d4zH185uFrVk>7TeigVO^R<3h2fGQbL8Ckut*GgN}jXzt*ZVdHR54ir^2 zj%5q4o8|;KON{w&v5kb}j(&rQp=ml;h@$Yk%C69A3aj||xI@=gLbI2CXjb0u*2lvc zJF;V?*@#e@CxRI;UN=QUukOha@ez*70*9?nWyR$vYij(4D2P#Ar&14NYsPMqn)VpR zP!8ZaGduI3>61ID4#OXUiitump3U46`cWV~6<-MJj_|XJqQolS zkdta>F2%ZxFjh@s^?24}2AL<{?}Y9tt9A2I@J_#-zWeC}YkKlMEUe&N>uTJ_{zI1} z3b9GjQ8dCrakzhG8O%Nc=Qn||vF{BR9yOcJRMjW!0VdmD2;z(3+6F#Z6>j(uVIbW~ zHA&Q>#CtB1Zgj0e3Yp)Ngo!(_(0A(-&yk{@Vamw_b9__ExD!Cop)><46MC(BiqZ^` zqQ%FMDXxcvj<_@L`YsmoZ<1QsnbU2ajCNJi6lVZoti_KtF~`gT(k7tA&YVVHqO=CZ z>C`!QbxNsZFBck3+4wqzG4@z$pSu|)|KI=f|Ed+3BU1m!J(09}i-jRTDdD#1l!Bmm zEjvL`WtshbIFdRAn&f`sTZN2bYUVv;BahYlL$&9gW0`wu;vJ#+^s?2s+v|0}QH+Hw z>H6YX(}&&0^zU-rLjzx>E@w0+O-3_x5;?O7wYgC>hr@PZzgE8+-Dt`0-SZZowl*tL zvXl=(RHv1>R&QVa@f$-n+EiDxp`w%pHrzNK-LLE_N5dr3iaqJLrp6A$pXOLokOZuX z{2hS?O?wV9I|H5k0_}eA{QwFAtp3n0nE#kRLGHqZK%65+%Fpw4fnL#6j6mA8({UAzkHm-N~_)EJYvFGY-EEyy&qLOj5emoJT+v%>72pmiFRYe&{iCC<$1k@s+BNf)wJ5BBH*1-)o4uy!mQx_;x4^G3DF z8dKcbm$Lbvp>cAZZp?L5`%_yc35(2r{^}|_gvh3?0dkw2YnO^mn2);t4?2Jh{r78`li{D$07j}wwjT&QfH>QO zNlG1Lq?V;A;Oh?Z93O0!bkt%mO3}DOiAAPf_8fn6*wK4z5^rTgizfFQC|*vFP0SuVq2MD?JS+D`qt|Du2}z{p=|Mxt zYLL#?PdI7GAGoV$bbkmfHu?np@G2$UITk?st!Uev<)4$&-!`YRwwXFeIpNX7U3f4I zb{KH(CMQUCP#z*8XZ7Ai3QcXobJ-A_dqVGPOi=Kz++B%I!g7Eudg-lvo0b@%qs0)c zULiN+Saq@)b_0K{qB!@v6Ny0N8tQD*%Q`-`!@@aT%{!ek4xp$3wc z$Z8dq3Ac+Bv=7O1f0NUQ4LwNpY##*DHoIz)Fb;&m9mJ0K_fPq?-jH)o*}Smm=+sO~ zz8WZ`+pEUY5nl#Jbvv!60S}ub$1Hc?t7x!QJ*v${(`&c@<;ufp(y6fwJ$B%hayJV& z8&3O_d_8&U+Ofy(U;>ONF;q>iC)GNEGVJV96uug>Za`nW0L@7g#4P9e=Uq_;Pf;Rf zSzXP!BeZSZj+(XkM0D1xG6K^+uV|m-^Z`hn*XfE{-4qozCQ5ezJ{x~ax?gRv&1b-Y ztF|hO)o9FIp2vrFPb^g;+llGpTlBOumUG#G1a|7~mVw5to z3D}-V1|a@tjCDopm1-wp7@{4UzEfgDk~u1s(|nC?r(Ig(rZ-3-K~uaac1d#$>e?1> zT7i{DsM2qbJxXDujLUL^1TYzj+O%ryqVI+psSIROW0~JQNgnmq|FwH+2HznYSJH5FqQ+v z8LIJAtU!8*uH2SL+Qn*{JUU=#%OBK8SU7e0jvnFq8802;YUeMCeRf98Sx*MG9(_fg z>7;k|G`pS*66yn{Q-c%4T~DSs>T-?omuP9>2Y{kxWvy#Wwe@jW1ftV~(+l^+*X1;a zy~zd4ubs(zTU~j*f}}@k2Wwao4Mni*w}Ipp_m2EFP_UWuIAcZ8gE3MnjMoUKpl~FM z)dH>XS9-h1Wn(^)4;MDIl3s+~DCNh&Eer%dmj)m#ir)J%s_>>{;vmB&Llj3w6-!Kx zj~%#>FwkO?ZqKqbVR0-@kdPG3q9kA(kn1}srQV6fp7;f$4)-EiAN}X&<6~^K`iXAI z)ar*f|BIrOe>?K}4RnzWcdBuyv6x{{TvZzodDvTyzBcp*8`28NU?oac*gF+n*H>PU zbyloNoRiWXur{=Vtz}a&P6hh2?jxZZn9qjD8Gg7PJFzi>Qi-u-&wD5GAMAh&WHlc? zMne<*&d7f*c8a7;Y?>W1BiYd>esKaf{7&9M`5x1vQGh;0O1nRJ5FlFv*^--MvC4|e z(9{FEBgE7A15kM!%8eI{z+_kw&6(|#}`HI(J)+|70nXkcjHe}d+6)1~mYI4RAhWWukB<@@NooyJzvUa}@-^jo{tUD&=&^(C`X60}Ds^ zCTIe$Fua~^vlI%!7)X)CXxj6%iJT*QMEJb&zhI%zdyTe?J*|EMtchgKI#ZfD5^fF=XclpSChQ&7I;9fi!~LvW zL=N!bArgc3{tf61GgxGh!K|N?ps8}Go+IYeqls_<6F}pCJW0@zcg)>SmNk7mm^TuMgm{5C2Zq8QGi4LGNYdzx?saPI*KP50hKMS0p4~{9>A#iHTHC zjk!IgE@8Z5UJuvQE~?g)@O|^5PTqbv{&4)UqjeM)dsJyY}PCloM!l^pXz+r9n~O2rZVs#i~MSQ%&}k0`O+f;^@}M{tE$^UYTO%gPPIM)ds-3?cU|(XCqDTq9zD_R(`I)t zTo=AZhO9t@-lPd2IH1Rq!hT6~aN;Kp^fV3#$?0$(s$iS|_YE6azbx9s9d-P3Kk~RS z9?s8BNw|It;y8mZ6wtw)T{{9gzB(|Rq9|LZQ5h}Q=un1|vW~8>;+s@dAIh#=_W8?% z?)=xGtaJ}bWP(c`#^U3Hd?VhujmObpT{IZvvkioJ9Z+=O=K89)v4IM!xwV6o)S=h- zHmd6FSh9cCojf^E@|(Fd6AqtIb3;~}XUCK@Wa5D`odXVy_S;wc(!B}do9G#tbTKQM zRa$4a3LiD14T(c)6&pmHuLT6ntc)7_w3H&h5@u`ytnvm?d&9ivuCTDhqJD?(A!R4AvRwd7&73% zEgrY=-`cxx#gL=6(Gf;Y=D`mtnP!sAX0wkS;p7PnH!z|+94S7tR~BD1<+(dl;wEJ~ zdIDJwfr5sBELY*wNpRat_{sgaGs&=RNCdgUYGYWQIgguGsg~*n5}+}V%%u}rr-&S| z^C4DMxh~FUPArs>tM1+w%D`BNEGaA7p)Z{WpP3{lvS8~w_&7#E%6KGj^XR3~Q*2df zy*f#ra{)AL6=@IJpIPI9QtIHGxgIxi`Z^|ASBAtZ;7;Pf3SBGuFn1^V=qnU#+xLdC z8Bha4HVSZD%S&FQj`AR$;Qw3<;UrnA+PG$ir65(XrVbRt38JqZ`w}3FefS)B#V1Lo zU_A4L&Oo4lC*}rjNPt-!hS`&T)V`ZMb=tZj)0BN(N&aD&sPtR2=s-3bj{WRrw$3y0 zT1;5J#b~@ocqlA<95(v}o}YA>l*P59h8RY#`J$pm=!4?Cx6bZF2J;0+lguUi8n8%QMb5oXqE{8B)72#2+h4 zk>#LRH~YC+%!isc^R!gQ*6w>x?d_^^N=18P^>nOPMl$YOm{8wCS07C5+~9By{if*`RJt7 zeeExbmeOsTJ^Cs{FKGPX6=)!-DA-En4P=iQF5Rj};|Dny{7OLA{wHC1^-y~{Vn3b9$I3~_m#vaR;&vYei(#3tcIL$7=rzcLC=rqtURN2}xq z6@qR@)P}_gt9bOJ=SGHd878+EPujXzwQgIHak$3LK~DTw$C{fvxNRvYB=azWk@zwX z8{c${0B}70#l(-69hTo?Kt@LQF8U&G6?#dz+mD8w>j6KG#|LFjvv98F82j}fERUd5 z2Es2!^sG*c(yYEmO&FELr|Wv*9l#EXm>y(vAf-Oy=z*KFJ-=~1SJYmRKminClv!YZ z0q_->m-j9fZkPdza#>?Q4 zqTR~!Pco_5sO^q~5x1$8CQc9iJ08{=-^QA|@LFS^Qz)5$_RKLc#{P*^oOj2;AcXiu zQj!(GCeE7y1zb~zb2oyhGj_M0NH7Tz?<0h*w%%BvKP4mYL85laEQ#E!HQ+o)F>a$sz@id8kJDB~tyx4@pSYAmB# zTB3$Ik4$%_!KOk^nai25`Xl?BH`i0LRU}S8%oSBJzbbY)d8hdqQ34^t^#8uNEvwcU zd;YD1n*lC*%Uf|%)W*l{Fp@(N{iDx?Y`(SRn1QgN)wPwHqS(`FcMS?dKtv0%v7HF= z6cGvrSCbAH^v2|w5=I?cs(4?lRuD>FnK}6+>6I35fok?5Jocx5zLHQid7#)jtjHeK z1!~AZTCWriEUcg{`E9_&ku}=dyudYezA^SqI(EpD8X?igDYmI!{btaB1$<4iFlQ!*?D@;%Qo&+*N54Y6JSjFFx=6FP$;7E!%Xf3RZGf*SGPFTec zRVO{6l1@g23^)P7?nLDcRh^vrrHVu1^3M&%nr!TXkJ2!Nx-ja*R3Br1HWgLSB$v@I zJV&oNo=HxpB5(^!^&nu@ehFmGsH38K1^P9t`ol<`xJ5Utc3+Bbl*;Wfag_V0Y9m!d z#)C?&;_{3~R7Md?A!`#=SACI&)b|VoiWonYaGVcAa=4g(`P&yGk@$KzOe9RC6xK(W zX6SqSQgl6m#$cDuQOLYYvcuFOq{~`Sr&xP>sXJN7c_RkxC=j}KJ8Z{&yN-m>%AL-d zyamdc?lu&MOAnZ87wlEj2ee$x!q z3yjESV<85VQlDy~vMPavdQBKSL!QZu%g@VndzB$0sN!_B$K5B(%oWCfCyz zuk!tWIIVQ{F^pUyLSg*Sttg+A`XqKP43v-1;h>k17bEI`I)l!mKsBF|;Xx0`bfvp>S0KY=(!)ly$%lZfNSuNg<7$5;VUi_bEi zS!SXN>qlL{g_csjt)fbcbS1Bem#}MA%r}DDgzMJuUrY1A3n_H5AHs@NNS;=6M2Jjs;8zl;-#1Z7#@pHDcf6 zWbD9CUKcm-$LtOgifZkowMrq4L~%+_p;j!FzM(V-EjeUGM)+6lkSff+q?zqN7GeSd zEm)Y%x+Jqy=bm3zMwxK6KH-rkn>wY%Vsbl<<#{~3ow;+v!hZGNKummnyy2+cBxJP%$ z?XX;~_dwC|s>RW3pw(ssoKoI%dOh9$sfu^PcHE$@x*grjN2q6JLHG9&4s z^mU8h$N2dX)le0?{LMnvHiq-dGa$LGdU$;JPUNuCJ0!BBuo5ayfEp?6N8_B{5(e`i zV;SwnQd__w#a0Nu$>U@P|Bqd_QOHPsYaBU#GumN2DSUDazdif|%OX_3S?uYCJCX~+ z;C9lcOHom)Y;%(ID6~fGMfxjvN`cjMon+5G!tAeNXv`0Z4?04Tzqr#;2uP#;Br%$$ zsyHrn3u2?#X#`T1Qe~IvLOIu(jUWuxyUE+U*>CfSZDXLOMF^;Eo)rKlMO4mm?3=^T zyKY1|bTN{$3n+_%?E6LCPN}RpXSkBE)o?Qws^bM;q5wcAG?UoH`12c4_7^b-{ub>W zMZIBNidKJ*4?ao$$HV{mNPaSLZtlu<2#Y&r?K-qg>@*shr=nM-u=;OIo!~uctpjgy z-0R3#-ZXhQMpvSN`CpkIrpAW}oS6-P$JXdH>D(&~I!m)`WIX4M5yq3w;UcfZH!`;1 zI1@thNNh+`X^VWx$Hx|-ln;vnu!`mo>TRECiQ6sSboV9F z1vUBULM2v7kQ*^A(d0?>t_bo;aXBM&=1yl(F_V_8(p?JD!?nv*jkFk@b~3-CqMP|G zX?hqvoFtDr3Glwez(wX0L1HY2)J%g;$t*kVq&*{s9C|kH`X5_ZGw-eteovL#2&_A4 zw%pc~I+E!QP`D)H#Ei&oG3ogvLKYXt6RIZ6Q;SSuLiNq>f(}`&6<_zWUR!##143OWJ;r0x$vZMgLm$VO@^%f z8jk*JBOX-t2x%xqfaKqq-0d9|fo97rrapV*M6%;01||2}w%EOeuwyma>$H zzDPLkT0>)E0*YT^S2f)?@;g~W%9HYcc5hM0pVoeXq|@nOkm(_p1GNx-O6X7m z7B)VM(WTpo-x%BVV`zKwcELKP4+HBEfCSqv<1sOUe{*{{UJk2}HW0@kkV)rVp&W*8vLPfz!*JvvE8N}9K-w}?$qRs2Jsy2=K21EJO zhYG$Y7b4>3?DF9ZadXz?o0c9%&y6zi3_FvM?ucOqG$VEp%2&dSz+;Mt7L3}Qz=uC! zaSQj^S$ejT&%c1pA!89wp|NkZ!H^v_q9DEEf%cl36N%qW%4@Y4+CI81iVZDxw%TXu z^1M!W7xM9^KRi467Qc80s*n8|CJ)QUzWb5-f;Ug;+ez{xJi>pQ?_@ohQ&kPs`1X}BdiuY-tv2}l_~2!%Q~b}hT!}H{ zH~dR%#7~!^;EDt>}4$Y^M2A!##5*p&87_56B4~6JW{`M)8=&E}N3w-8ELT{09G$V)g8s%s4 z+M5)R43QROKG;B5ww^pL#B_+m;5Ex|`SdAo;i#aGUNGBHjJsLDjig~B@aF!H+=@M8-|^TQ4RJ3z$0 zjnB>?Xh>ORw4YgaUBZK_87SEKjsl#PpeLWV7Y=}kb-yeAoS&!%U($0=QBFn55F%5d znJh#++ZUT`DSlcu*;1-YORm_{F`h%K)EU99L#GeyV!uKno-J)M@lO4~5Epb|;{6aN zpw@OoWrFp-BvE!*)nt1_AsNJbltl&SU6@RFv|Pge^W~Bvbboz^BwUSLBcBAyaB2$v zU(!&^c(gw*RTRLP8`7%rgi|JG?p#K*EP(?>$$RE8aJjh!MzDj?JRL*Mou7j_Uu1*Z z0^MW&KM%+R@Ipv>Fp7Fmv9Wv!$_%x1Up6^tURdNGUUgd#zzHphU<98CMNCt;vjK99 ziqM~&=ah9I_^|i08&qwane(Rklk?W5s?+dG#xKr0EDK^&`{lN-cDeH+|13V}r{aTr z&IXObk2%9Q$O?lv-}uk7k`msu`?K>l1_A(e+k{_se)S+NkSAGaL~@>~&d|b{o)w#- zy$(+jdc?&eO&FyR2Xy>0-}qG%qq8DQRT4#Dz+e)Kgs_}+C56O!OnQt}9hIf|f=hEH z)Oonpm$wR-C9;fIvFSn6FPjwOALHd3_)ucRlKiNV-<&auW+~MQK8?ET()dsN!vXn} z&rndOJ)Hl5neciyNIcnARc$B_hN{)2DuM>RPFU|iK5vetgqf^KnhP?dpXG9fOvI@A z5~WJHHOD%;PWB}0ERj14m^P!ah-8KT69z8~325}_r1(u))HTOjQ7gJC0d0HrNv%{S z$~H;9UKgu^#+x}XR-oZ*!DflpE|KU=F@69NOj$P5njLI3%w*`DB+p9_^5llG@6p)Y z#(Q)5jS&V%RWmpqINv3sS8soONHuE5?rwH%wfX5axyK><$vno1|Euxqn8=08{0Fq^S+l*ci8LkY+PMMGdkEA zO5K5#j01aTLIA>h7lE)IbXy5)bK98wJx2f_C-B%(LMlktVv0m?~9X+kT zNwz76yD}-J7L@|xRMi9SP+4Kx#yO$5i*0pno zeV99W$^}{t?N1eqd8Y&LUsqJ?JB2%r1yg<^Z=yNO6^8XwS1Am~2in%`Dz4{bj!czI z&~&-@>HD|eJ%2khhxqQtL_fLP1664r>?rbMO=baFCByU0gJ)(z7)(y`GW_I7E@!uY zmhp9>QYFjQuKpbp5@;yYvZ!7rq7JTRiPhE6J_pZpe<$nt3}euQdV#fHq!4d{;wQzq zV0lM1Y1PK0l1Zqd!jZ6;O{M4}uC6SAu354UMkj#Pz=-{sPQs6eu3G$r+<9@?1z`n--g;LWSNDZ4V+EqZH>*iNRHHdM{ZeoxUGmJ zSPpcP?#SWCjIv2aTwt-$k_XbRH{%J4q)6Mh>9yAgxa9C>NF!t?~Yb9q7%y2smD$dF7UZ!Q8Eo%N?-+u1_h5-rOF-`Pd;%Ql4HnE zbX>}42;Ku1i3*~05}<)vNHtkqV$BG0K;pI@*{DBBjs!Jn;$@1CCQMYvy){P0vAY)F zDrRrNA#~kdcFx+fPpIaaMa9IEZbdyyIdxfAq$8u^hOSR;6jtK+dB1O=jwPhVg>KgkH{Q_4+4a#6 z<)^Z`DosAev(1ygbS>s@t}MkxjV;mj4Vncoxr>~NjjY~9kagwNtCTtBBIuAD8DJX+ z3_p!(o>-KL917Hnt0dd!qEvF0?9+dj5ut15EX0yuHd*J(&EmsP@bg;S0;eB%;YtE@`x#imX#@g?C<-UExqt7~~8Cr$wuY zm^rOMsel#+4_=cXLhOm^w{X)GPVEp{C(0v%^S!GUwBGD#YohENRRmbGR%+m1NXd3B zVSNT81SzSWVk24=13iAlTfl0mun!caDrL_ajeSiD#$T!dhOXc#Mj7X|h4dOV$y_Ep zt8mgFoX(OT#BsrrC_3FbV%_p)Un&%l;|44eVkW9-u4Y=&OsX<~yZiDV?%(8L!qpB)}I!Ky4hBLJ3PuUF+d5yU; z;Z#nH0K`P8L4Z9BcjN~v=5}%aoF`lIWciYYA-ifH`b&Mi&Nog= z7`Pe|!DUY~(FlroJk}K1QBgJu)>O-pg7|jJSWNZSwf}aeA5Z~U;z5V+#Vl_@2^3}! zpsuZ-f)t3TTVMjz8o(Tc{Sq7# zCteT*NkMKeFrm^qx6fqCoF^A;yKCkT9-M=Myg$Qu*9W-4sR$Z=a0ZA!-EH>gCQeCl z+_&n4%|IHwltz)4BlXP;{z4tHtv<0q70Y;1DV&D~TZmM89^4NFDB=M}fq~2zLKRv> z#w6fd+X`+I%Fpxw2sg8%4hi0JCrF6U@kd<>8UQK?P#8UA4C2}VTfji9%UPN+ghGA} zjIMv~99cLOm#a;)tj>ORWO2}-)C2qt#ybYX81e0q-I(1STQt>kYt(rw2%tec_Ayvy zGlH#=c7B17%Wl7IIF57SZbhn>;_FeMcOjk7!auZ$c?^qSc!;?uBk?U9CgMHd>$9=@oJXCSCUD~vw;GoqhP|3AP zR&zmiE?~bcGFj=!;#tpe&(k%CQ9y^qE7X`hOCBb)j7f4v=6peILAXd>y??{QI+C0P zA$l1}3nfG3nJ%=X3&$qUa;(Hulw1wk98R)OQ-HoG)>x?xH;1OBI#=C8Nr2~{&@zr_ zWbq0{z=W|9iWFiL{lhG?F;yRYxvCC%2p*f?M|vPDQ{DkE)nWX zB6p2x@d1R8%Dp;uyH6=wA$0H`$NZTcm%7|QUdZBdrD751E|nP&zbGa%u3*q`_X_5NPbk&h<>L(ZdUi_}X!mC+?NLGKb zXspx{urPDD4@GN=>oO0rs0hH)EZx!P0MulrK%a%V1Azu1J7B;7^p5OForQ+wPymPF zxA4D;yo+(GSo7GOKC;pqoPd}iJl;66EN3~(vuh8dy-V1SejXnVY(@P%iZ}q(12Hs- z5#1O<2M#)NR_oY;iFYoUcP_W>ZZr;$K(g4wp03X-I>O|2su!k3h@F_>i1_I~)nN|z zA9cQo#i42m(N47)dEQma58Uknwjy;Few0t(@uFM`Pn@_&u4m7Ez!}4CioEds7UBWJ zw~U?c>_HTYE^LgQ_$^R9L|w#j*m+%f!WXt79vI4+w>Ck8K^&n_Og5wMGBY^?L<74~)7ruC-?(^kJvA`x2u4zxb`_RSBX z919k2mx;?}JcqD!Qv}Tg)zRSIO6E|0W?LgX0DH0sG*zN2#N&_YWm=HTJM+yN+DX{s zP&D##VJq6lMzf59cit#e@0$=1_jfz#0rVVS#V!ZIKes(zwyRwYbNNWz{F-uu9K|Z| zM8MTWJ)`rQjgKofbc2pk7qVN;kB{!Zd2?U-T~PCwwik8Pwi_ojHbeVUGCD2MVJ94e zu$S`!FBF<)Pd$6Q^uvn)yJIV~?~T$U&e9FYF64{(W|N-l2wWLiY`6eu6|x$`XIk?G!u6~|6<|^UzW77qP-eC}}Y(FdQ zSIm)=_%~HsL}<99@tjlu9f|n*+{>REYW|*lH2{;@Fdi}c`b*+ zE9-I)^_d%-MmOvEZQHBLT!35?3;Y3XxN1NdxGd_bME}R==-r!>x8E({0xaLWeg4gh zmoJx3|M>jrcW=Ib@%sI{qcPF8M*+%q!*f$qe*jSlAMtGuR3h{ZPIZwyid$<4D1kBU zw%$I+>ugDXDBg5oidDU+7f*&@mw!NU?0WkB>-R5S|8VmD#rLlRDE9E^(3Mv;$l&F$ z__vc%chpb}fde9_q$kO5T+mK5v?2P6H6tt+%d@9%D8?u+H^?|Nce>Y@-i>_;BdB`6 z=_e7aY&#t!TcTCaaH4#BnsARXYyn&%3bmpkCjN~AI=mam#N!JkTdOfrTtvmOZW%6c z^U6@8#M|6MMG0I})Z?CO^b1i|H+{=iea)6$(lR7KP3s3pH}NDW)ITXW$q4VkKsX~u z@qvtC%ClJ@?8VX2FeVIMs!;ylY1bNsyX6FiZ(e-!J%`gzI7TkV1iUuArv7W7fXqT_ znn-@k{}k|Dsr5x3x+~^fC!;lLtN4Vd`P_`>G7aDiGaX6%hbDO#^`B0K^!0qs?T z&O#|q7_@|gAwfVgV0OnP7n@n0hr3SGb0c4I#A+)l_u8tr(wEMybKnt^`qJ7ZB5l|E zu2gSxIau$(SdSUdL@ik~Krmn>&&q z?pjSaf+raMjm4eSZQuDnh68vyA*z%E5lEGYDDOntyozQUjOH0Ri+jR_r+41Bz?dhfH5QOPVs>H5b%O=M!2-TSd#Lv|}Si;XCaZP z&P`K_)3Q4pGAX;FqP9db^W3NkhN*1o!|aFGZ%%%A_xza?ORQ2hbzC%Qn?hK^6^~TQ zJ&bO^s@O$|p>pRnkXmyv+dM@@1+eizoo~|uBgdy~@@7*U8(7aN#F}5^e(vt(RNjUQcT9d7T^iQMvBihsfOe0n`*8jWoU|FvC7*)XIBjOemA%{bCI zT-(<%%&d1SViR`InWO6H$ALN_I?1+tJIN8Kx;cC<7uOJu**a_{8D!7KcyXITJ z4U3+?@upgvZxy8h34`?=p4Bq^K)9jdrDzF=jeKy?AvU6t6j8{+#1he?7klt<7)3m} zSYWfcyP+5HdYQIvIWF%`>h@qpEh%01YVNHB6E8$>DcLW0_$4yqmhEj|!7@vmWG&LB zBoh^h2UNQPQ5#nOnSno%Sq0{q3r$qDx9(C`y|<{`yu@I$_cboxzD=V6dOYRK!Hts1 z5BOQd6)qMbyA;JW<2Jbi7k-r6)!*26j1o;KcriYte50W_BjYzaD2aQ;#o>MM4-Q#bEi?9lcgVD- zV4oZZ_soW`w)jikXr3g;e%#$iS#dkDBWeum$6GSz_`4nJ&Mr=pqBBDbe~h2}lZuXZ zGt>wyMP3b~I~jp_(98OrfgC6>j*dk_d9kB&yzd+mJ-596bAf0n+|oiL*1~8^w}`Gi zAoS~9FML3l{hbzFcFw@SW*4@_fz&=QT%=*10($W0z&KueJ>7Y(vkcR?x5S0)FBu7g zj2`+r>}Z#wiV7hq2iUQ&UJz)rD3ZzFL&Y|*9G4bCwO-#li;=7rlOYg{_!QyAHy zM$WsGWhOEqxYbs5te)#;jBAdmOnF5+Y`h>8dfW%xlm!p|!=sPm7@Rs!Kc2Mhr1^U- z?&N|oOR*Ks{vi7jUGTag;NKH7K!}q*XnKYefHE%yFy8sUs=8G{a;D{ac`~4LU&$}+EL&0 zQpq(?>ged`>899mxh4nZO>a4#H<@N0O&C7A|Wsc4egP3ile7T}}>D zG);KpT~DY$Vaxfgj0_z4@;Uig6b~QIk{4*170nGB;Se>-l%QvnljIPsv6id@!cDO) z+6kLUws}?{84c9hYMWUv158Nj8bIX7waPCCT-po2svtT$f z2+y<9!5HiZ?|0*4=;0VHM#7fgQG_04YG?bJN~tgq=gDb6wV5UlPstd7h)IEoUh3R_ zYgBpQ21VDx-(A$Qf*=n!GtWSB*Gx22j?=;P!g<_5AXBla%P@d~gh`WN$ z6-|$*K%JL}3TpU%vpH#7tV^*MS)n|`7DJV(kR47%?&&iOd#={&q_Qg$Gd*alB)zPF zkEenz3R+h*CQjklLTFK|UWAb+@a1oVxiWpZNOzN$9NY(jD;6wnWWgG2%*>`&8lUo6 zOA-86P_>t;TDxO$4_7)0yU3{%#3Jbnj-GN+p0r(L9-ryvj4pz)}z$aw{b~jtIxL^Y9XwehP95J4<)TMYYLz!D492 zKF>d=7o*?!KdoO{!VdQzszFlV3$WG|m>yJvvab%La5?Zt5`De5m z&J#=g;PeO_W+F)xIeaNkBT2JM)M^@gdFs|PW*J_yS%kd$&Ib&J!*yNdhJQUTg?OOQY zFyqL6{D)OwxatbcLC>Tq(*ZO3AOYkE?kVj$ITO9k`~d}*EprSk0(1vF=2KVb^<7SZ zQ-<`&Lx)Baa;13pYZj;scN?82HuYeURc&<^R9L&hug7sWBN`v&$*B?)on}clt*|`T z16-gY3->-+joyBoEY|8J&!P|++jm|>H-eT{|A$FrWH*jasEy2g@PFgH%v-zp@)N;B zWZ!iWp|}XFIj}LF-Wbm+or$s2Jg9Oo0}0^y?B#ZlTf{W${kZC3(FdF%=mP&L_;ol# zI}&mlsQe%1xypkKO9SpiseWdi+evcNYm+$gJf-31eAsxiYyUB9&|jOG?m(Lwg`=%Q zK5~K3_5xHKgcoHH*@Dt@9vjgSS4M?8AizivJ?ugW9eo!I3GIk;;n&dyTd4a3)<;N90dR z6`mKbrojEs7W>5}Z?bQSGM}D{R7;Hibwm@(G6D!(bA3wU6=zOFWPEM1ku8A;Y5zK5-7ngpVCc@_49EGjbAUs|6k3LM&0LEmt7PP7CT9-CpY> zxw>Ns)SnCOXxFI<~0_2#hMjn%0{lC)0tnNGi<5e!+A(in2lJoh2^& zN((a+P~Z`l7Nnw~Ou~#UG?JPs0*aqJn?I`2IO0^AeU5DdHU|8$0nchoa8dNfNg0M? zj$m}!+fqOy{STIkzin8fmA$6}7F8k046CxABk$$yTwI^kMMk*}VN+GuA%*Zz=1Er1 zspzXNRj#8S9V-@WnG2~=6jlVaqD^~xmk|iN^pAV+LMGBHZ-xDE!n#-E#{IYS^6Qe7cQIAfRyA!Pv8q*6P zlbL19B&^9grD6+YFEEaf zyDRZb=sv;fAtkqa*;Sj(zLmrp>v|I@=M6XaROKJjmifFO!AVrgHH*)|&E4eG2&tuN z!ZBE^t2zKTCX8_Dr$Ahh5az+6fg?T6!*R^pp#s8VR@RUsd>S&t`R!lw9MQ}lBz&ld%a=M);#z>oDF>qS`PU0ay&#&Ff`P+tN6dEq)d>>Z9;ChiTq)1DjU1aTsPVem>p!>&q%=91SyJfCF;b&AF=~18@}=jRc~-IdB{R#g z7Od`gfSx%THgnZz8}i3P>&NR;v17bC)4{x=*mh0rMSrDk+tHt^x>j8aG+Q70*ce@; z?jcAtQkZtQ1{H=MV)rhBvg9;@rW^9c=-lOnEkFx13ralI`MJ}DQ6oOKsBJ<1!qg=A zd$z6MtU-@o1yXxGrGW67l8jN6YwT_f?<_+A_d=jRnu?;=Q4=5u5|+*?JoaRmf%Nuy zSJX}cC~Q`S*W~-rqp|QBi2;@KKwXL{h_Oi4i}LFr|4BEP+KA?$mV2BKuQGbAQ2`wh zfkCl=HRAgkqyqG0&H8|nq?bU|E3enNQ|>05qs+|<%uT)G^ zv!`~gAnVq|-ndK$^>EE|T)fC44~~k#gS=g2ct2CXW@5=({;a0L2Q{ic5!P_P%hpnPJ zNf&wG?)s$raX;k_o`zAhbPj?dtRJ>@y4#Uu{!}2V>|@ZRTa^9&BzD+@F9Jv7iEdI; zvY-oN)>HuRtgg~*g_b4?ts%FA;PYP=P7b>!=dI zz77o~KqjgvphJ=RR#D+pQJ(V)?%gq8E&8avYc%kmzJECRm=2Fq_S4yprM2#$un~C4 zF;|CE4lUT+Haf){(09!Hk4-GOU}7`%smarODzu=N@spyAxd>@e&MZ|!1{0w~1z#Y6 zEODa5GV`MjnVrkXCLvny8;T56TA-~x-5d^)Qv6o6_FLf=Kdx)_Gc7WEbC7P)3+9vE z4&;3f7pCh@4Yfou8Qs)v=hAvfue%GR4Mx;K*(`y3K8f1NJN7IM&1!$b<>AKsnKlVD zd@0d-5y+-f83xq5+4UxHPlcD)YjL!#f`m}9u(^QHFOhi1CN7XLG7R1*@rhoh*4I@kvJDw10skf?BDbv zO*-72kq%HDw||KH*Sq;$C)7z6#sQ(qUDzTRf@ydA1ziYv&{=NO6+FvVXd9waDI~${k40pC~wGxLAP;G3l^L7?n!=tgMB`LRxgs7?7z-DuD??wj1I3QMg z+EfL^QsS?{V}q-C#JMAq`pa&N4bYwoIJ~endN=tO`!x}b)c4l!X32}P*{|2dsvy?N zv!RVCb&`nqzG09S1&~DcExKV+8_TVk=7Z|le{!1Re;x(b^84^w0McX$P5l@HaUmm1 z;RR#$R{P>lHAUcNPr=ogKHGWn$Gpxb7}J+gh5l2i1e{{PQY(ndDcOBaC!yi4Q0--_ znhbT~n@21A=IJ-@{MP8UAJ()S%lF{FpzrNu|8wOH0hpQh{tIe3hI|57A9f)uD&PGm z&DPwbTiqIGZbA%E!)Yx7?3Pu%Eccs@D782`k_(?FGGCm<@}qEKcUN7jo=#E+RmD1t zMfb32FOMoR%(%Bt*UhpXdj!|R1&l-X8rT7wcCoO?JT9xLJ_TWyhm9?_I=Mk-BOi2I zgf61}MLTh&3lkTk?%&75Mh^S87tZHFW0~)mRR~7iRY$n+Ajd83Zcr9(NKF$zLHEP) zM@*4||MwGd$DxE?ksC0M-^ytH8v{|h+a%s7pZMUzjny}E9*fW7iIsi4rPB>ESXiCK z-h0DgA&LPut;Ydp@s@z((_;1tV)Q%3bKh{gjW=&!eE;^v`~Tembt^0-nvm?=@|vGA_$DXmek!q39~${<)h_bWL2 zj8AbeJ~0%UmJ&l2!5}ZH^XV?TXGqHzeSe{*=zhzxO&f24C>6C+hmO>Wh`Y)1yP}MO z<9Vrk-AC_~^;|^bu4uA;?jK%G_j^5ywGh77fEl8=baq-d7cn$awYTVRA7V(uj5JAP zj-A-Bn8@aT5l2@@lFheg*aQ!-Mj#v>$bLH-R!Ppjy_l+DM!E6WqhdU4Fcc6A%A<_H zzJHhjGevCM_ZMMs#2W=4ATU@w53xEbO^Rqqet#(reKc^U4gnz$c-QeX7Cgbwx=k*e zkPFcqg6ddM%S7PCm}Yc6T#5Aqd}m}S%m&eGj|b_xzw<60`?^ZDh}H7`w3QrZmONz{ z)Cf^3ngJ{~7VKlI0TD_5?xC{=XETg4am!1DNeZBc)hj=`^RJ76;ciCZzD1y#pZBoM zsXo}civGbj*tR@9f<0emp@p*w=BT*=Z6HN2bHN zy8;t@lSlZO`-k~}eYf%OCkMqljDwYKRG(1q`X8tYWL?}OUVUCwaR zL1;jp&>Cd-$#zYS9>na-kiJNNae{FdbKtOIQmR9J;$sHm^o0FVjrFfDni2MF&Tyi6GH?E1CJsp1bskL zXULQwBI<|i#%9IV95U1~I zbSSb#ah z{Gg!!8%6RPB6q+LP3dOn(9P2=Me5~bxoeV<0QmBo@82%py?^`s$fp@1A$H0-R6lIx;gcbZf*BKzV@L5>=lL=jmK$66L9h@g%NGXA8D@@9w!j&BQ ziIdPa!1xz+3Rsw-9GLLGK&{jw=|^6wvfPhC4P^K#*<)HA+M^55q94CD+vWKurF)O# z3y*3#!IlXtyzlE6x4XsZkMlU z@e>MSv+{u`luaz#n@B=QYLoLQ8ynbIh%V6E=dZqh|9tt~kMBD88)g>*FU|6^{dteM z>gczmlJn()I~SfycruPAy+&5q?df2LjDu&@t_#g#v=uY7P=)t^A08qW*bz@ra-I98 zCCJdL!Y{g=oIG1n0O95_&)+k)GEogM8bjtPsiWlngmbQGjn}5Eu41FDs3ZtHNzus| zSttXzDV<0RO>HI54o@czxGi%>FTu;BN91J;MxyG|Obo z6UA{jnc1Qi6pu;~3{kJ8|@~M6FJbU8Jn3A|WBcYtb!u#m0&) z%n}GiR~%=~;c-CGi=y4yqVSE#L{JN#5I^|`!N{%>`AM*m3u$GaE|;8E9Bi~CCS_g7 zuJHxyGAlqt< zNmG==hv}6ZxY-h&q=7~8=QNmWhXDpE0h^sZ2;@!PQgd=lmc)b<$}O8=F(}MePl})f zD{MPd){r+o7xAK;#qr?z$+s_`C$zLiMd+rRtMs}F2OwufLnTvaGbw5&IvbuF9*WHV z3+%mdNeSs#YbBPg2tyU?0%M^?iE&JElq8J`8O~zrUNpQ-HCVK3whhqCsmdIkkS?0q zqr}M9!T&9%qsy8fpZ;oxj$o9QNGM;;vucFs(RI4$&mPL80Dl!9%i%P&3wwQ zC*n_a=haA^VG75TAa1<)I8@}S$A>Gjv&1#V(A9EPAolXT`0mH#ysqFEJb$uQ=1t## zWd zUm2gfTSCSVxZn9NO=%VurWNbIpwsYnP|X4BbB`|%>`V-B6XHSjF#&WL&It?D2L^#G2I zC|Z`8__R1Y6{K5%vj#t&+?rab0$PV~Mu#i*zYt`ba_XZ210Wxo+%^tT?e#E*DqlN@ znGT3-yRl&MWx%hcmcJADZz#|7$}%Bk_1nCaS+HCyKl3~A9iX2^mKir!#CtqY>NX_2X`NDx$1(3M?@N$MQ#Y-9X`a=_pCxt;vn! zNjQjQk~D7|MB{d9%tOEXQ8k0;Nv=6GFhK}t*b>XhIA-$>N@}8vuO7PTOif93k}Z~$ zA$KXc*>Ef>OSs>G@|9zAF`lI}FGDtn;EqsA!bn$CJQc)&RkgdGR^=3Xq;=)K$R<0Y zQ&=bZC%^t zNOT&k9gsY&e&Q-om~b|sRc~^c#Fl1#W0D;m`^o@s$Min*P>Mo(Je5%^VZimctBXGh zizPR}7Gz(TEEpg3`4#7`(6xZ;fRkKs<1OCl>{dWI?pw@jF&zjiG(F24 zKo6!*_(GKIWt-Bkk*bBx1p}dxN;PsFV}jMG#Y<0RsMA@#305+>fm`$dZHb| zV9qfkodfrqSSZy;nS5jsOzQY@f$<@t*MpP_@JKv)k%vx7OFBLUt3ER(#ZMw|_ssiT25f z!stfwb9eMX*9GaQxqwDgw`enV3c`zUUtO5-ES@6d-}2E7HGEMPCccXU04AtlXn{c= zOB_V+KnhiYIF&2fiQ049EvFpFYV$ct#_^1x(E-8u!)yVj*?GHNuJaUox%5OKsmM($ z_#Z$3uL_h(FM)4jfWJ>m=L5PkXsT%sEJ$Qmo!8~T|ramH8;~W8X;W-&Mz8sdsl{gk(l@0kLanq07zdQzX3UQQbo(4Tajs8wP0ICM?IW#of2EaL2rhmfnPk)RE1(sD)Z z5HekK9=XW9#tbdoY~m5*T*xC@mVS0nMo|ZSkPOjjW`zL)Lu;CkW4m~*Y<;?r>!3nh zKjj$2l$`oUV3x&8%4U~$e%SrOD4eINm~?`}Lavj&ZG>@@5QbV4bN2obvOA&_GYC^-sa4{35R1TxBNx#n)ePW=|M?b*JZ z@})}f+V$9PB5WZp995UB* zW}QY{)65~{c5(}^)8yuUftwXB9oz+BFcuDNwGuGT4C-VuP&7KYv&qd4^UzKw19ai8 zgS>^;KGfzylnOQVjNGv}=hf7G-9W>7(NY;GDgVq7kA- zXqau$P>2twETGYYtNMEn2U}Jr*3X9L<)JWbTmO2L!PVpUg7GfmqTOFo0PE$Z$@a+(Eg=`x1( zB*+(<53yEI_vgv`%q#xRA@$2*QfP3i-9=^PQg2W~c0gIQ1E_`|V1_OH_!h?46A=Nz z@gb=DG!#GTbWXW~UvibBLWDa?whfvgTV+z3VS^?e=_kU}X_D8I6Kt0q%-!bksI50v zNy4^CEfx40tznd8DzX-fa>_L}1ZcSF!+6qR!D6nI!F3~Qg3H@sUiXR<)D34=v>t(b z2%^$W$xBh}#d$DXSJ!WqGNSZl_Y}>@>l?RGiSn@C4k_cXli^ErK~BU&uK9{P9@lXCC)xNMv3VwH}L>w`6j)#9AgU$xh6wN<--3e@-Etkm)t2}1qiVR z)fukv!ZA=tiHx2un$2X;m0LKUBX1#RF)Wk$pPI#4Rc-9rW6>GEOp}^hHK_RqYfh8& z!!p=Ps3$PxR!y;PrMV4liq#cXG8rh%4K2|L)rD15>HlZ% z?U&p*l03ov`^@ega;mINKnp;Vn%>BpQfit+kwa3_-OVgd**C+B4mwP?$DA&DP;b_8($c-sK71XO!KVu>t=h@7& z;NK_KEQ%@0vfqA6*0?-QHH4=^+LeyD$Ribw;k{vq8JH;{&w9fWJ218Xkj3~6r=QTi z3GL(_PDa+IIvBsY>jCubtOc1n>Y4Xumc#RXQ#?4<9O#Idk+9FEk80qg@$f|Qn|sbJ zH0r1ZEI{ArQNe9HD_Qj{WYbCOYx_GAO+7q{)3qH^+WSWT{K}!KNLG7O(5hN(c6W5$ zJ?z_4i2?O5M*yf~q^;ouF_mgBCe^6?Yrqc1If}G8qRemMjnes4*El8ub;C4(q z!1@@TIpz9}KrUv1*IDdVpncg4(G)z{MGtb?vKmTnx2bkJI+%!y1j6L*!h?Bq@BUsu z=O#}DMJH%>i_V@3$>8cSc;O#e31&E87y;igsCeDmFG z3e8B*%?7xx`p2SAdXYwd4nO?;Jfy>;Z{2M{6G6JO)^t;H4K6ptTi&!IVy&8iNj6(% z;Mv;T?y4u7UG)q~zP;O4lQ&QP@a*Z&KR$ap(h{Qe*0cMwnNPd&^T0HBXXSDr>8{Vd zdj0I#i&t-x6bgj;K zpBc7)nkXqX?~y9Xx8u`ubmcKrN95^>W&%doh4!OP8O)2mppH)1^$EGY{%4rTIDMYE zZc-K4+Wywpk-<7}Acg z8EwD!(DSA+qk{}?vEBBxAPp1(YX~7>By|ZvHQVyC-rtSUZI*T&rY{U~%XQrOK)6P7 z!~yM#^Q~&VwRt=^y|jRC`-~h+e_!tF26(DPE0Jw=uBjRf-ci$|8#T|&#Xy1=7@OTv z-`7zRskngbl&ejq`U-5zTq4j|)Dme@tgl{x$a{0OUi3s}iG^Al^h)m~?BR_dL0CKvJ;q7kMeLzxK%~ z?91SoCM^nA>GzJo%G{yBySiDeHfx*Jp)~)#2BT^+)HVlMoDhq>Q9MlB4MOpN+>oq; zAl5I-2BI|!htm_UvzlZ+>v&2m*$7WJkz<0ST%*yDkRLlGk7`feWND3j$)pC)bhkCx#W9ODgW>(d}? z;jqJjZv-0(umhX_9Jqx;J|}5LNl0P%;VX{&q6~W3%#!X74(GnP250s|b|c7}H1apa z4hH*yxjBYQ{XX$ib<208r>4vV&;cFU-}Cw2)k~}NoBInYn6n0XfX<&LpeXo6yP1OE zZJw`on{7qlBN}!8wC%Nz3-vvE+|LoNX7jXT9u^mY{EA8A?{uF2u7>~KGICYEuPjlA z9V3{hL_Yd5ZX#`@m0rW3E!nd?=0>drg%}w1v*EPXms%x8J?vAXB;RZ%ov*db5y>1F z8^D1kl@_n56oVJ9e|b0tKgTdlB->ZsvS#~XPHMYE4WgM|6`h@Lv2kBjPTyB|q?*bx zctvR&c9zO;j1+=Kr&^35D^GCM{NWexx@#uIQ&6F0yNdM-(zE#nu^_HK z^jTX@TH7(N@Usn1sqzOYI0E7KB}HWS5e4EzbQHh+Dz%!qIk zFw2yDG`7>&Ft|q%RUY)R7;x?yS)FP3@Db<#!$Tg)O&iY4XxfIzk6Y^{^htGmexsne z4c-)C!~t+7Ru=Pb%_ZC9+%y#K<4r>1VoQ4vnFB65)g^o*1u%m6&HB^Jl9W!N41?i~PG8lpzH|@$W(M$1untsq`2}lzY z9k$>oZ}L-jq9=jUFhr-t%Pr>rsMomk>K!k0_eHZ3DXCr2tjle4vq7G5a$vXQwsR;@ zvzgbJ5e#>A6Y@8j&7rw@`z+3*^lF@^Y;7$LvDTGx6vj+l4NCcp*GTUdr2Q+6Dqv%v zlP&`6I*{cd5tZ3ETh8GKs>dEhb#{zf##XIXo=af@d$ss9HW|IebqGIUh#%@cl6|Q9 zXu1os{Sl`p0;;+7W<5sEt++Nklp$m7j$Hxu9yXb&&FEq0cngG#({SVxm@$hO0zFrl@s4-aHc5VWN@P5ymlI{k=OF3l$`lQh zXCpzj5cFr53V(GP%)IN*-3pfH(Np4_3BoHaY0uPAKW&VVS%Gd4S`dVfCt@6HcLQBO zC?>M~bJ})IlNqKL+%AF}m!uJc6#^||w-?fY?W5mw{dkdaZoK&N0C%4?h* z5tTCNa}9(3g?GNj;A)fcO)sZyS&R>q$ly)&-VzV z!Zh;oR{s1c-^$?@>A z)taLkvRe79^A?b;DHji=#q|A!rfpha)D_$1F!dOPVXb@AJ3+^-%DPPWGw>*wnlzF1 zSF2YIN4?M1S<7!xEv5{wtNF^DW+0%u)?QpbST>Pi2*UOTKOA5;4yUwdtyvQ3v#N$@ zK4Dx}z{IkJSoXsJZa|U0Zo6p$h5xj8I4NGifHSxpn^j<*by@1GN%8Bh+!nz1PdH;= z7%>jND;lJ=+%^;}0Ey7Ir4bOt<^s~o$;$q5Qal0s8CGLTll=;1SYy$-?jD?KlYMXo zFosSKxH!~HjF#|`G4OkG#gEVNj2FfJJV|O~3oiUKoUU)71yR@=HWB9M*^?*v&i2Gr zz;ZM7O=V>-e2Y?;1&v8GAbD-b_PN!lk_?diq8uP@Z8%1c#f9z)qQnWSj-0>tZb`>O zG1%X1nu-sd%l!fTr?M)9U&8{0(*a?G--6ll2&>xI+Z`Np)=TsrBSio86kC6{9C;7 zPRYoQgt`%_4!SBROA6XQ*qF{D9&O)Pv#iXe9=hIbZi~YjPu*(qi1xwz50w5;gdBM5 z_)0OH3%qP58=Er(d~wGc;uM6Mxq?V?&j9;P9{Mg-drz3>PzG{RhU zSGWt7i}C~(Za6%ea#f9SZ*tls%9eyYLbnhlz@L(!GZgonhR(=sbHl>HM9_|Ny^&-u zU0xP*3_0JG5dTO8&Q~E0ueHu6yt{CF%nCHem(A))feJtTG6|+QflzftUN8=96q|BR z5Z!I3FvfJbYwC$=UKS=(>#$C;9^IeLA&2K?OT6+%2d>{mpQLZ-i2etGICHhT+khg= z^g>aR2BZ04_?d@50)Alk9u@6)iJKWF59Y*N#~ZsOBVQ$XVtM_V&52G}6k7w>QCmy{ zXATWsVq3Fh4!a99C6CD*BWv@<(_wL8S^(TVC^%P#y+xkmHshe}UGu1@%#_l>hYs4n zF$*SoOtq!!T@$>3)$2LWwCy70nhq*jUoH>krh~G;9F^qoW(O~R0e!`0Pnf$^j;^IjVs#1D^S= zz&kbIP7AAZ`(7NhG3w&54-8ywr(__X`p2ZVS^t9guK;)`)QXIbk19o{j;*N*;RVcN zeB*3vmIlxc&g1XiK6@RbLL6YHQOQWr>hM;YD&0!IO00pFSDm+en0WJ)T)aSzlR+l? ztRaHx1#{&*f7(nU8%Mt(!0>%!bBC31XaaV@Q+X2(_-UQ~bq$w>dbO$+b%`lB*>Jly zcRsx$vhfBsSu~?OzapTw&6HP7t|yehgMD#~>;Vgmz3z^}-bEjb`)cF)%f%m!+$X!P zBD5p_q5{f~Z38-%KEmmtJy0o`GwNA`pe`I7xR@22cl3obU6QB~X*2D!`S+7!tWO2b zM~Lwkbjtv(XxhT9LY6eED==bz!*&#`XlD~4qjm{WDj(9gERDor`$hGqAVq^SXy~1; z8)2*kUgy??SjE?x%T~#9^JUkOa-3ZXnE(-@eSnOgT%Xf-|83UREI;Uy@19MpeL~XsHc!ZsZv(#gFa?QyhU3zKQL9~elC(+shUl8uy z4GNL!BPf@R1t(cDSaay^_Mz~73^ynz_U(B##2^(uA^n8}FI{j)inF>o7*r~v;9AoV zV8sEdn#1mc(UHl06*TAP#@1olU^bG=b7X;oZmygs%Y|pjdN}zm(a4}S$YvX)TTxm$ zKrV_voA*Zm48|o~3-sfkPZkgJ3;>L~8o1W`{FMdOggxax=GE9>J)?1-(4qSKR9{vZlm<2zqX>7y`#f< zMww!mls@T2ox~oVDI_r?TMG~-!wFolO^TF|i|Vq3Bxg0pUzba`@F2Zxtdh0Fp{VXu zoX~)Ns!eD0rBfZ%s<0q*D`fvlj9~m+7{TaWqq*+QP0A>3*FWQfR#TV%g9M4mnss|3mf^Tv``Zg&GV2_b@8OOR~`+}9>bX?m4dhwz%Lc>TD zFrY)Cd9?mIwJgXRC#xHGWRj1is_kLYW$XLyN>2(JDPn%w$;KfQpEIW@eE_K3Mx(u4F)Tl z>@Qcp4em-udKg*NdC`h=%aLflD4QypSWS?jpV(F6mHd{P(TwYUg@V=|f=VB1XBqUI z{@TdgMy`w$ndv`I=vSi)LZ3Qececl(19n)`u7z+eb#+BACrbisG{a&4x$RreDGM`J zun3ppZsFj1$}TIGzjW?dKucOI-HkRXR_}pQMY-G69{_MtuVA%t-{jNu9h6XvRw-AF ze=YhU(oFc}b}A#J6MXab^|Qw>;IG%u{#A37LdPYUnv=wwOm`eupyE&KU)H={Ih&QO zvD0StuRlNg`PtL(aP1KwhG(lVw2A1jDfu5B4q83h?o|_z52TAe9OjL`{I&JWo(lhT)I5Jt?kXy#T+`)jZ9y(i5)tyz zFtpy(i;p84YFU}r0oz**0OPygY|ORt^2Xd%f(k~#+8ZiH910K*15dR=k|Zn**|xg% z^L6p+)t6=Dd>LU{LAs=|eg|Xnp)QNkOcD@07T>@6Ij-_zw<#BwpdxNdbL!B;uLMG#dMUD6R2oS{ex z=fo}!Wxghto9&%O?#L`6gv=;&9S;w>(vCe6e`;LVmAXrco`Mf4x9 z+2ssZK7}6vgu>LX?%cXa*bhRVsc1MHmUOWy&clH?mpSxU)nQ61LE9Y|#2&B17CHBz zXw8-A+Za(Qf%yX&l$Yh!+u@L&7(Ksy_*MjOq@fnqskP*zswdS%bo1T)bD2%T_FChP z#c(*&8wRrhFy0M$j&M;{gE2Y;!NH`J!8^qO^h$x*9?DAIqKy}Ne7L%*FKd`xb-K_& zftco^-cO<`qA!9sz1bpB^@h4b@A$WXCqABMC8|FMraYR&1+Rt6DS^rRf52T_O%ybg?mO6q&8T z28*YHfEO}R1Z*NE;c%twk2t>q$2`m-_w?kiOiABAXA&P^@6O^}w4+?E;NT|MLJFF_ zJqk0L@b-$QiCI;?p;}S}6Pz^Cg}9{_;!AE}>P~5h8_&B!ws(` z;-)qq!Wbw&-2k4ThD0q8W#V{G@GJMwUrfhlXg?13qETi!K_cQs9o8|~!fCY~Kf`AU5LG;Xn#L>TdBa-`nXXUbSX zbAxhun~kDwa=Jf>L9PXHNF zl;!=6nuXi?i2OK$BADYxIe8XrV*#Ld!AQK{FRJUB0N6$)I?2?=tQVL~lSquB6-pKh zVs-*l{N$u(p0==^pe+Po)>JKvctgr1@lhCwYSovrxG-B}QD0qEprKB{tBt0a0uL*0 zxmF`qf1!7RY7^q26yu{^Geeprmxj;yvR%4`r z4KjOWIHEi>rlIfGn_KgLwaT1&?Y|oxrkJ>`r05(<5Ur$3&SdtGW}Hq@%l6O8<$NM5 zXrkd0- z3+c98?~~l)r#5SySOkxiG5E9k#bANE{rS8za3gkC$H5KN)kwe$ikk)dZW#??g@byd!9kx zJ4|2aOe)yGj1xj$F^g6CF{7|NkjLOSHwCJi-_HlZNqKWpJ0nZV_1q;W$YOMjOv&GO z^{Y^yg^(IkkaodrEJkD4AK5Gell$?eMBMyDs}NZnZa16e@P3F`KciEk+wn%d`|+rg zcA)#*q&m6WZ?2K_e*{ruvbsZau$1GfW^+-KtIjAkMjB~6&UH(y~(=Bm@;TXshp(a!Aium;g%t=h1f;u?(( zyL^sGP_9A=!U-@Tx-B_(&DP9SRemUIG`WEi&AEO`cEXx1`^8=ybn80{o80XBq`z$8 zB+Aa=gRHoT56&k}l@4QmP(pwOya~*bk*!^k(x+kgksU?hfYQ?}(nQpYkMmu%WXb$M zpW3mXwP9k|kX6*g&=;P0BhAc{uM)ayr4Q6A%mcY1!+e7(M`Ix`do12d9c zJcfGDNpK5B5EHCTDz1U-+$9{B7RKS3iQ8lkq@9n(aGVY_8Wg5S-FaFmV>}yQzyt#} zWjbU*5mY82x~+7dlbdIE&CmwQ~UYWgLgAORFbAEcOu{8*A_bK*7I4H1>>Hn)y`jr-&L z`&J1<))OVI!f_Fn{YZnd?^_ZL4V06vSWg3ZWDK%3gaC+j+OorjDd#QXoH_$Vlc>z0 z8u4PsxNSK4ji{Yf8_=87=)&i+%UT42(MTjNxRzFqI^uGBQp2XqWx>`Nw70>ILU6LA zaglj2Qsi(MxuriG%@J=>BlBXIq7)4-tIOR zxi{_4g1(#rLF^B6XlmwVt?1|G2-t2mzHF`+L7@l|R<7~#VH#Oy-%ksMoc@k&&e6AM zwJ0Tp*Zs0@HA+ce&E;WHnti*yaV*$c6;Y`WB=HOh>%M>WGczbwYos}04_nPxmvXlR zrv*_lg3Eyk_rdgSEKjPi1XipaLwU2&*T@yh&Ls+$@WJM|A< zT2=d9ecAXso_tC!FtE=o25C+f`$>B&1aE?pgTv;_!9VBM+e4mP5lLBSTTX>QLp3{= zY8d5WhOs;>_fT{IrG{GQ5Wu^6Y~GlMPx+ZTMncstYgi&?2jV@$$G9YescmN%PUZQe zrkwFv7DLpp8n?WeH#@+IMMNv8wozjhj}b#GVdldlz?^2bsE+m*GC== z*mb-5`WjJ7^a6SASqtP|axY?5?%crAst3-yWi<~5O5@w_HM;l9Q5MrWRl5c>=PeL$ zBJp2HQb8yK8TLXvosc&^pqMxSkf9hndG+)78}lDC66?#malKwtADM0JQSpCYRky{9 zZxIm&9~%bB_luw3yag-nWwl_AAWS(j8wNWj zK-p+XT~}*_w|QQ7M@Tvf=yH(|(S8=v1LHb+`qdx4SkS0wCy7wP=uMlzF5Fo5M*9Z+u1)^gqgv%)O6WL9ORhz)VFe)<=>P3=28j|Sk zN!jf0wpGr>sx&2fba6L_k53j*%}Be!Z|`Qv4La(uh!Qh3hgPqE;Ty3JGT!Nn#@#RR z`WII*5TR~V0>)zm-Yo&OL}f9)KH@<^-%gD7f`~(%&}{HbwBp|FC~ihWvx)D2nTyIo z6Gg(k?!0!7n|vl5%bZDA<%d6jpKxW?)F>wBz)fz1{AO!kDe++5>{iG}cCp#-%~z_+ z_Y+SE0w9CUVP{H*Ji^usje>36)JV(DsZ~JAp-`io#vAETVG$FCQUQ?E1pq6YApA~1 z2-5jhxhS`gu?W>TqexxtBd1MTK=the#fYae!`K<~i7|hFfR8zXP>ivEYE9q73;Adx z9ba1OhDaDgnqS?Lml21Zd}M5pjys__g^BAL+;@Uuvlui+iH}K+hcKLY;Ojm+!Jpbf zQJL|CrlSK9Y1x>?*01#LzCx~>ZFwTeOm{QN5|eeZeM&E+LJ`o-#M&Y*Z6y^uVxfNc zh4M2%*~6}CGxoOIiRm+rP0Z2-4f0S4QGtbQ*|#P>3;GRuCy-+QrqXenzM6~84%8XU zL@MHPYnPJ_kXU=-K^*Cz$YNV>t0hEO0lma>Y5v`;3bbq&aV+>SkJ@m#H!y74G@`sC z?qg1(0&B7uAfg{D{oxm_DvlWDbehlvnZ=cWl&;ryAkZeOI^z)m(ojGU&Y3xy>CE-H zBVBvkXZ2ntethJak`s?{4S?-C0+&flcg(Ac+t>1*#eCdD%ws~FM(9W1*SiYC(V}%4 z%L7S(n{tZxZz^s-)ahp}3JAE!u#`?F=~VE*P^T$=v7PRn`~Df;^b>m@`Xt_+wOo0m zv2rm_NT_?H`)IYu<`1yc>6C2-TFbR8tm(LIb|O8Y91gNb;M$ob!dH_|idKPyH?BRH&J6QWL?qgbl+XO zlJ{lWg&tMJ3Lw{#?=`Te{*hpAjR44MkQ>M_Jpy*{iUqoQC<_viOzioH<9ZRQ%a~8I z*3|FloIjpUvcajr;e~eGEev(sj ztz@RJW%neXAJ5t_v#N@b`D(yx^I({-bl86NQ`QJ)qpdp7y}=HPdpc@=*G_Yv?L|K5 zJznN0N1)Q?IfWz}{pQ4$JsD+z#j(=;9k|+}@G1mZQ$}t)hAh>-O{m6Ks!Cy=>K%?Q zoM18?C)0gsjMBoWq}x3JbS6C4X>NBY;3xzc?x2^F%CMq&MSe2rNH#Bo-sebLk6hCN zJ`}k=>>bRLAg0A&OY|saktBNFW($})G;_ku&ta4WHj94j;CbP9E&Ch4nh>fgEARpm5v7Ka+?=ox$?E=RZAv^M_|o=WpJ= zeDz8uMoKqkYI4I~<~$l^tX3ZO3X(?;^lH^4XHlvc!C2ZWj7xHwGYlQ0yULb=zFQ?O{Y({6!;ki!Yb zKAunsXT3x(0^S4`t9lO&S0i;jXFazPX~tKTkx8865I+Ej&^@{j=`)Lek0sbHE<79} zk98ww%Dkl1FKt0(EmU3I3y>NR0n!5bn*pu0Fb`-os)Q=etU?MeMmZ@jTijpvlXYXa z&VY-KfT5imPJj-^;th4)ML9Xn)wQZWy6hwykd~qQFAjkekUEW% z+;BSdy$Fc41Hculme04ldKFd7#cdi-TMNahRaMiYUS3~a7C0Ygn_aL*AA}1jw+)$# z^NY8A5%-Qv)9+zvpDenS$Ab+@ET!`YE-Lv+ZAvpTcM+oy!pKx`2WuIGlb_e%6NUTK z)J>VQoAEjGo5p$zaX87?DgxH zuiL9YLp%~`shW5{ksLucG`+M%acE0Il2IWf@}gB3;#xgUh{J?9yh4IA&>R39l}Cbp z?kMd0c^w`8$!W7ezLf1|>u$*J&W73OTwR5qW$j`{$>g*{4%si)W-q?!Tg{G0%63Ik~7O2KZld7xuaV+CY1E8 z%SY2&qY7f!4(qIVBIPUW{OYj9A|pRhbQ)tmYr9))G^{=Zjr+Q1#jLy ze*HE`C#WlH5yQpEAHRnPT2H50oQyR(vWNV}`c=prNdyfGH6Q?~geKtRxCAK!Y{Z!`u=WSb%Tzt@BX#sa^ z)^bs>oo5@LYKmt#X@D|}v2Dw4EzDjY{5a5$!LWjEA3;!lViEi;0zpqhX4+V<9OaQU zWhnVzTkW?5xnPw)r$O;>tEP`9}s=U)tIfyf% z`_&$Epcb40g{^!xtChneA;R6FEtK^gchg-;tNmPLCxc(LS$6r=A_RKWua?0FoEBG- zSr)0qcBGqU(`4vbYqy)f)Hs{T+dgg8!vp-L%q4u>VWA^Cqxxbi@3W6UW@^)<`3N(# z%Tx=^m(Y@F^C)$!O$~z+*teytmNf7!iz5kr5%!;xW?5eb6Hl&zks_9dITi=L@&ulY zX$CiBe{T%}bcOVLLTWY(Ywnhrn&Ty`BpR5Gi*IhodP+^ps2oQ5#e5hZeVG>S0MjP% zgXu$mCyaN>_mx$L{Vt)TUbTbUPhtm87*n}7N67}WscX@etsDTo4qeCyzDS}_eJfw9 ztrW3e?`%IVm%v@l{;_;62l!w7x$awAD}F##U0Sm&WKX17@K?vT>{QI;A06)mdu5h9 z>3BP{I%b*YEwxz}70u!-Q1V({g9F13l{Z4UXDx4H;4D1T>pc|E$X?K`700b=??xvR_dek!CGM?E9u6w`j%g=@v+BUdVKSwPX zUclxgu+vdD=y#UIpN`t`y6XGUkE5<@uhMfK#7_OxWez1WTZ)DEz^URNTqe!vmu`iY z{V0p-;&7cKGJnk7MoEfP-yeXVal5}k)quhPah?2TQC!}Dm1mLVF-Mr)Z;&wC{C5C8 zWp81kshk$SnS6CMLVt6%9^j~4DK8Pl-4rTEzJg)aH`BV<{Pbjpz5T7D3PVqOEUMgF zJz#2mw<*R9GEbn6&J3lBf`!fB8l+>6J|C*w zW0u3#H)Z-CF znM`XyiYs$ACW&yhYmVC0LDuCHf2zE`-c{FdLbbJMn03M*+kZtzOsnXWV7xGJ%^>`j zB&wqc;KUT7xZ>lSZuHIsXu`~auhb>SmvQQ1$44U_s!3Wu>9RgEFFu2xzeSsjIe7tQHG%(_ z^VXimYJcpaz#r!pIlr6mzXte$sNGK}7h*zhN%;=}b%xpxJB)A250$@q#EBSfgiR(% zK!x8hTV+=Xp2)vMYe^_u)3jLwei7n68b!yGuaBTX*cxlwWBwTb+Ft{5I}m1y zh(A*}sxyi&SR*OM5uqAakr&>nE0hp(2_XaA;<%n$h$?APOmo_G*Cap&0G&zH#0-}X zA8eIql?w@?iu9F2>=f7nJ%-un6B{7zRTcp{$7Jh=!yXP!Q(rJMG@!GfJ}D`UT5xkVcj@jwG2PU^9`<{i2-H^G#zp3%N{{5}vd& zZ;)-9d9Z^BYViGrApeMPqJRQ(n639?BdwLlHM;=UMxFzE|8Afy#Si8306d_;m}V-b zm6#3x`n96?eI`@2031%=LM%Evo5`-ohLCv4*_K;HTb~bLQ9{SGr5U?Q%!ctgi_= zAAR=t1^o5;*}rDyL7O zw;%MeaAC`$K8Q7A)e$=Z+p2o+W+2uCxl-|3Jo|00%aOAUHXV?S`8xOpU5J#=Xo(%7 zanH?9oZ%q{A#B-3k#spZIpGro3BiQoRtz}&4I^vVT4>{K?D;*MBuU~$Hh5QFUxjfr zjwH^o=uFxpId5*JI0T+7tE;_oukt?3>%>81ZGH?9JkG3D058(6UQ_{H@?$iB z3qE5TdjN@F17d-RS$BI*pjcnYI!Z|-*z75<39Ld!V{qbDiVsYTh!MZ4)mT8{ayC1- zXI*g~$P~YqrL=%oIG56>h zBjj_2=njrDyRDE9hoTxHb#ano{h)-w@VDQ_e6jdACw%L&S`Uwaz{5laN)x@=T?^l9<( z)!XMUp8sEu-#&l&)3kVuw8CiTGWW?LUpMHQ-V}=d0jE{2UcAXfzE zdEcRB`-=t$jNgw-P>UckyM*1cM}qV6t_hjaG4uRo1@2w5^`mMaw2`Z!4B8D~wJ?cl zy$`sWJUsO1ifb9<37Gc$Fu=(%sQFQ0rgo3=}1?e@X5`S%Ed zp+oD_gz(+uBWkd$xsF4|2j>-=!z8@r>)A7vLYf#~EkgR3|^J3x4_+~TdW@zk+GS&-2hQG68 zn57|F#pq$?e&NL~R4Ej6PVl0~*VyBsH&u|mooVSWt9|Kme;iHllMIBd&SJDCmao!A zltzTbC~fYx=GIo@B)&VWjg%Od`~B`LbM=is89wOV>VT}z+&9~||8Yq&BRP?ytkkp2 zL^=E-gy~V_*Z(Q1{+6OjoRZokWq3l^-a4* z#bQt1jUS#Bzg23;u+J+(*ap9HWNcWCnx`Q1wtf_an6h><;iw4#E)d#6 z5t%tLjUgN;nu)(DFz`wu(}@ROSaoBIKAVqlG6l|kIP#D9?IH7&XugYK+zSWOIy7d7 z`CeQtH|1V$J6T!EOPT-A;fXjb!>Cr~NCN}J;KAg<_~GP%-W?q(hHa58mHxc`mayt$ z-Rvn!Kt13hTyqGobT8+9{?7a192Gwka1s%@?JN@F3)Y$N552t!(`+1|s55ZH$X<+! zlZ9EM^`*4e){7JE#3r97z!K;(M~6x=kU(aD{M74esWEEh+GCOGiOuH zTH(DlZe#8i+u{kbja6Jht>K(*`p5KruCL8orA8PiQ2Sg^!Q*dVzJB}cX}hyZ+^%vC zz|f}^E*VI+YW}6Cs_pNG@-aKy&v$7*&-^i+Q3~d~0-;$-n;m z>}OnCUI6EVVFGbXmo~GIihnDfgE`r(IJjG#F>FFY|1{W*xW1vE&?W{9528PyubmGf zN>v4^RE_F{fMEbu+QfzK*q=yc;5)AbQ{t@0ubvkeY)sZL%%dgzfVY+*mLPY&BFH*L z6V(JCczz{E#9o$0h*=c~RrD-MG2v_A!#0}6A0aDkg3(0aCFlN{6u)xPFCz>I*9nRE zT)8MaVVrF(7-x|_kxrLT7|HuyG_B!LD!72fsDQC_m${N!)>~OTkacx3z}Ymfvckq6 z{E;4<_w`HPnbRQNS9cMLT_|%{1fnk~Ixo#?841m>tTbhj5-I?s@1hK&-_<+FQ*7pE z>QMB4acnp`lJ*6GfHZRi=xTBY{|`13Jm+N6wvy*hk1z;?LuU1itkjGF9sP;4@)O=d zrIpbkdNkx1Ww1BKWUQv#$}i7!IF(4?K*))e2b_f)P}MN_zuJS%pt`!MFO5vSybIAs z*|EK6FmS%YP*ml+K0P9yH($$rVtQL^ljHx!9PE&l0Hp!=b$KnJ(+S+-wuVFo*wE8( z^Gp-_P?);1E8T3bwd|3tIvOMSLk=h zC^(7XEFYHE#_*DTDaUjWw?JIVw6SX$+60G|3ZkqOrQg*Z%knYY_VH>-ek=6cufw6QDEkAjU)Mu zsi<2s|1w3D5#XD_iK%T*U^_5MldJ+Nspcf8bFwa1RWf{{0B~sP?NtZ4dm7C>8*}an zw80k*$aH&dEC8D0kc2jxYvc1H__~0Ak&G26uE%t872m)5nUeN_&y@hpx4Rm_&($h- z<#(qO+_H$>$pS&hQ{}QsBsNP7PD^$bR*0S9#1yEK8bLdV9b!L1kg0ckV$-dqVr#Kh zQ;1sgeWVTO6OI z#I+2E=}x%ZfYEYq=EJ zTAyrwHG+P72V+Kf$9VtLyIwm8;IJ;$CECth{x_`OZ={O~1DJ6U=C)GT25>!XSx~)J z5mgI)n8>rlDerB|s%_?71MQ5xn@-0M&!Smh?95!x!z$<;P{W*TLA#)DZ#J+q(V62? zjJmWkq5T#LQ8*hlksz1rs@#oN8#C6Mb$wZYx@n|KiOa&R2>ODA%Y#$3bk?RZ&dL>i z;esM~p*bWUDBNMsrsth>S3o;3I&57%LYr%2q-Jwfo3;_y{Gz%s$_wbi5Cl=}fI5uS zZGhYEa;t|{qcT0wlE5a9in)~xVc~ZV&XQ|7jrBJ@mz)yP0RNkhB8e2Op-coAXPQOB zBP2Mlc)IZB!2=A*Yi;E)*VTs_$t!T+C!eq(_>+r)CcVHb4tBG{Ith~ZT8)uNS_o{Q zD9nU@GI{m*=QrTD`|(FT1nN+86+*+9m`n36yV!k=4l1+w6{m-$hR zB(Y&KqM2W6{qYk9xrJM%DvGnr-B8*hBU?Rrs?$M+-brhHNL$f3F2v26ZXM<~ap^X4 zX?@tTR%6y&aA}r`y3fumL3Ly#T#yY`1e_$qi-dTYvzfMgRkSSxeYWxw?F-vTRwI#u z2E}vTImM^NOO6dhF=F;=wYw@W0gr!G9+rCyg@MqW_Z1%R6%z`Whrm!v0=Enaiad1| zoceGyS$V%bG&jhV#q%96YY=0Sl^95J>s<<qfT|3~byw=Zd9ao+K+_ph$F>Z%q&nW89@L-ly;&?~wPw z@7{a^)$(-DVS48<-GiZ0Z!UBqV_4D|%u@SNrU7}T61itE0S{f?KGON97-OMG>?8^8 zueOw@oXeSKwa9jx1IS!#sU>7`Ougn>Ltw~25|aa6^2x@|bfJz}22>nU5`?Vt2L!ZY z9lnHcnA^HRPyp&^Kq5j_V>I$E*A0gdkPdd7=d`8!@;&@*5pykceh;lIP(5ig@!77DjJ|E!p?r#o_tqr4yo5Rjl$T6qQ;S%)Q ztHUyhN`6I4C8La1DPp5sz>VXgynN3~Dz4Fv7-R2@D)9WfmsDx=b)nt!OL^pu^T1O< zw7=w>OO^pR_4Bwz0pS*#Ed|`$6dth4w0Lizc9I$S9BQ!PRb0G5ez0326f!5mB}dG2 zy3M%8FNwnff3x9oO}V1G7Oo5o69-R5*>L!9Bc~$4vYZCYk1hspyQ}fouD0kPx%04_ z7EJGT2|cgaGY3-=SPoNDD<4ZF0XlvxXf-j1PN=3&)8W ze<;>jOC$J-*AVr*s-9)iBcFVSK1)KDz{U!A1ljbM6Vb3R6p-mjZ%OT#xS4l*aX=fc zRwvE8X(BAj&6RRqaLwcZUjY?_>5J9XMrx(x)#Bs3Ic&FxxGL_d{p6&DBHc|*j;uAe z1fsX>q;m*z_>SUv$QAKAL0om$ovG{I1Gy`OmRaui6g_NeG($#51+N?1EHyf zM%u=;Aw$HDQErDdpc^n|+dRH=GN=?^q(w=3c$t`ix18R+q%}Ln-H1us%jU8Kq#Wtu zM!2AA&M06Au>N8nSJqUOrdbU^uirv9iLGaQKs(Pxb%~A#xS_>0E$W4-A9`_BU&Fa2 zdK3wmj;tFe(X`qzg2UyFr35oql-uSK7vU?N+?+MgR+mG)vWqPIWH4V{q`IB2QU{cb=E?~8jTiW8{kGFA+g9TYJ@t@sAXL~m zP~R5w8H0R%C_X8KO7O3s(56F;);&pTTKp)9t2L-olA;DEdOp+$a<5hw)dDUimR<}_ z8ldfIbu8)5Y8;$FaCuP~BJt7^F=d2Ak<-G#y3MnQryFr1_a*s`UXGY0r^S<>pFaK` z9>9Cq*F4?iw0LFyo&N|f3>W1p%T09;LvAzhf(+NVvKMmt$I4~U4a!R;#3k?Ux53)> z)s!;Je?*s%GCD0XrDKC<`JwY;(N^Y2d10!IJ0?dEP2AvXbzIQ{-Og=;AQ<#(Bq_OH zvOLz=7xTgoiSr~Jc^K_EH%RFQ)Mgofm~YEFNYc^4mrueKBRF*)^mcxHUhLt%8Kg0k zG6Zjy#@3}T$OyezVi;9ki0q@$i?Q`|ln#lH&YQJF%Z=g;pz*?+rH@Aacfzdg$6TYM z#XO7t9(5*pn)y{W>RKza%sPuZ8OrNfJ2S8s(=(MISUIZFUVNs5YZTh+W$rHYkVn5q z8Nsw7edrL9Z+oBz2Kof2RHo%!2NhUO;c3#X9tT)Hmsr0%Vi3IpQri@EPAkBke>soOc60!b4`*&wa!Z$Tlq4KOvz^am8V7P7$`^DyVomFw= zg=P-H0fUow_N6l~zNbV0b)p~*)06_!eu)QM+lKc`ED>E~WOesOmZp|}z1rTxX7Ljz z?)B5p^gpgGNOx4cu82A81zOIV&;7+Tblra0FEF9Ge_J-y{FjHR6q4{mkzORMUQUCr z{@IhD`u4hI6-w))6*;HNxa@KTjf515mk?5_MM@T%Et!VEKVa>61)^-t`X{#}g0(Oxa^VQ` zl9NqIwi`QUjs^jY6asI=zW`3id~dwD8dL5}>sDe9hbKq9AwLHr)`~r{KTDs>Y3Sn} z0j7Y=pbYF86T8(#*_Z|ozpp zV&%r& z7)xb9W-Wu2>jbN4B4{^U?xW%jnZBDpym&lJEx<3|J{$aSSntbvIE5q>Y;amsW@RJg z1UUc!AVJKhXmkZaIm4alwnr=G!^$4rJ~qKd4vJ+CE1Rh+1{Rj?(r7bPQo1k?E6p01-;?Htp$+kMk+5k*V{ROn zDMunW$yI8W)DbjH!B$nXqlO`b?1clie~23VfMJOmN5-#~mP2ZMHL-NxF}ehuC01(^ zJ+<0Y_-Ts`ASbj1cyPEfA_c_PwIJpW0*(CTGLwEo#Ob+6=3LmYl97t~c2i=)KeW2q ztjdoxb)mRLQga-?xH9VxGR=r6lp0DQVGSDbNJl+VKC2Tu;nabCOk@!D9i0y^udzze z=TEnm*0dyU#{dl7mLv>w7U&2L+oR&9=;MB))3^h$LOvF z2eWV!aN<}ge}oTnbs<#CV?yuHY3Gt}LqowWs1bd)T?&v;NJJfRv$``(2XSL9lhP|A zU6!Ehr;Wad;FAbryQFw#RrGwiCd zi-`sq?WWR5HJ!rSHF9K;X*}h&T0B|S8@ww5(t9%OyGx=lF)cVZ%T_>l!gk^VWolqr zZhf#vFfu5=%gg-%kseNpuk?f>x3%?u%bIGV*uz8S@T=BvV_x(M8@Z3nNI->PdATeJ z-_`)SzrH}KY11J(;V?57G}v@4Kyzn*+vDUEGa5A$)U#Aem_3Y)aV~7_)%ZbkId)HY!d)Z-(m3 zxUm-rcONLwT;It^@k0*da0J+~Gus5WY(Y%hW=TM|;wfaIPBpc&u~7@(S9hbrHkL%k ze!rYI$hbljzh$Gqxvl+ek&oA;656~wzL$xI2jOhJ5GT{W|> zvMmbq=UKPfiW55O#mwSoT}|jC5U~uPS4O&d=LYpGPcd`U;CgzWAY!)ovUoV^b(*!D zXcV`&T%v;7wga;}QQL)-BcfWUd$6UEJPEEr-RW^<`_%|?Hyesz1sO=j0J6!!*^A1) zy;qromBlkkS4>%qi#)Cgk}o69n@q!OQ3V+Cmg9esR8)DlU7rPuPHKE;VPsV6P^u`# zN=mpkQlhq_S;lJv?{xj=O5GZ5@U$0zwZGhK;T*XE26ZFS8peI%aAMd5TlnaN^G8`| z!|?~$EOE?8wkjcy179h}C_|(!BDKLcq289;#6X&adpOm@X7Ni$#m;)dx+B{JFn`hO zh+To!5g0V{z^&c;xxV4eYXiC00<#Hwfm>L<;H#*Q^kgKO^f=CYg&AI$glr z6sHxwS9Vo_gWW+Y#)4}%rwP~Zfj$Qpn+kOL(e8j70MsIAXHes@~5h;-{pvZ-38kB*q(`vqrEK-pug&5W@YQg-t5nzHCx6PA#{+zlfxNKTt7 zrg4bM)yZGNUL$?iUjtUtzoe7&*OLy92TDy#+~(Y^blL>SQ=5e<5s$;8{2yU>Jh>vA zsD>XmHVv)}=BSEaNu>^@cmv_7$?*Ytnoy)2Nhp&Tz>>B>cbhop?r&u_a{J1ZxUw|g z%4txZ8!*LY!C{KsYc?FOt0Z?hECV_;KH>a@41+nN@%=0qKA!$~eh12&N93HrvZ+ zksWUj*(=O%z`ll{NL#t)GA|c@JW$>V@IIa^hO2XXj4U_nYgl4b8Qm`@VGa($ua_6~ zvfkf~fEf*GdCX1vu)^C7wHbjYA}tyl!@zYUXwoFK;v(>AmY0|W8GvMN5M6=*Ib@G_ z*O}x8W~SeAvg{!J#s-I$b7IS3i&NXBEAjc$2I^{kO<$V4C&%B&RB?{?K=qG@i}u%> z3N!!iH!f)eJIydXAcyG`0}5fjk|sDW9tfUc!|5?JzT8z9$xNZBch!E_Bc};&6-5w_ z-DJ6Y@4#-Tp&LF=5wqHsygd|?N!uXP#|YizKFMorVD_u|*xURHM4k$TDessSn6}0v z%WyWR=O8;wV-P)u{fF3G{26r$N-?c<9w5)?1`UP*H#9^Hka`aYO`uX-@6%xlu`JQF zkWj@eKS`z&eN<{6sUh<9sAp6-TZN5mii#s1;d=7TXnF|zI?MaE0G{&W^)_m+Nc&l^>haya%whE zTmW3a0wc6)P=}_>my_bL1=B�jNgRcwPq5UT$wTOSMgqKXnHIE2JACs0SJcEG3AM zx7(Q*iZFNsKtjra5sT#fttT&(zntav*0GMu2kImTDoD#p^+y5U#|O#s81_#GC3X}h z5hDrHvb0YjW})6AREOmL2GYV=#xXqc%RBrY@F&_y1eg*JM8a97rm;=Nx-G#?PUw9} z0*89j)PJtXLmbGg;&;D`OGQ>RP+k`wN5M{pY~*IKU*qU&4IPe(U}%SWS}Rv4+y2ut zt!pzY)dt5-YzoyrUa#|KY975QGz_P5u$sYyn85W_Zw^h7%_Pk+EbE2D*7nf>l*X(o zy2T060GAI!m74(rJ0JzJ#I)v7YZ?!PMD`b#3lB{50GOYBJwJ_2ElazP_9PL zd{jJ!KXWy4DWiH;WSG0F9Gw;qG-J+=UmCetMuBMm8fKP@_Hz63*g1rCGPmVAT0H7C z%0fHE8W5L%3VcyE{^awFjN3g*_jZT%rL|2*u-SkB-6EVbE3W;dFk+|pZM$@Gd=~=QV(3R%8rAv3% zy+`eu$EKUnEfh0|i@gL0U?F8&K=Fa-rR%H>;1GB3i#rWEETbeofT0s@AtXYHklO-} z{i^q3^Y1`PW*2J@aAS7+%+Vc=H6-YlPTW9ro&0 z19UIDvoI-JBTLSBOj(idZ!gxZb8pDv8C$Ka`3V9LK)XueCIxcVFE^WQmIw|*a_4Y< zn7MbfFmCID7YBC1+{P@0E;pJ1HbioGQ&S9}9&dmg1wwL$hKGQ#%)Cq=K@VuMJ9aXa zk1(@1he$-=^ckF7D%}b-)Wu-%x`h?Q&+M9=XDGvtbhl&%Xp}XriaVBme#a|aDV9#v zsGH&3%Q`dCF={ad`mK{e0(aIaOnbjqyPtQrka?SMMeGUG!uEhA4DF08ua5V=)x{9t3s^+FJ92B zLqh-zi&@`pmi6UbF}THOGn*VK(Q9OdLD(&umWHc|V>fa^m5{QuY+c8vtgz83$UK-Q9h^@xc7=zy1FA_Gpqf{ zW_z~-9*nou$NjW$e^LtV`r=^ehA=+=#~)t+5cQ9k0J~n{EgZZA`^^Rb^C;KKi_3z4 zdi?E^k@>s9T-*f0;XgFrR2)ekYKKpRFWbzwOgoqY5Du${Wik8F%*9%3OZ zR%Uh9`}sVNsSKuIsw^50CII;`^^!rNBavod&cq9ZkJP?lcUsI#>|pS7?+}Q~EH1Z@ z0L@$!+^?p`GOT*rSt3K_5@tM;h@6jtL5goN+iEv=!DsM^EPsC+Bhqs-Ej5PU5?&y; zPA9E6#c1;0;&K#^9LJqqy^#r9&g)ypGKrZ}GavFZ2<7Vq8S=$0AUK3I@9Uq{r~9ps zhkygltoZd|X>PH5b1nYy$qUX6Mjue1cVrxg0UbVq?P~F0QB%HXit}^-+4J)le3@Ay z7Ck9r=$v8Ivbf;OiY2VNUEalPGpMw48mjC_b+15k%euRS0UMXM;6yP$jqDesv8qm7|1h5NbImFp45flkC$-dHO0UjD$2d|k`BZ%((0Ir zJBLkkj`V0&LmILRe7%8|j|!WRNrQtJt>hA0i(CO?CuGOX{%%Q9m>Y`>g9rx74HG2A zS_UC0jLC0WlbmRa8!%$Z;~@2>+kW^$ym24VjNws)3^~?|NRt(!2x551ER+hW*Ojt3 z2NKl07e0teZ{yP2OzHY6s;Ky;c<5|SHajhp6VMB;g<*`P#Uf7{XM zP(DT@qYmMg4hR27lGR2Yex_vjASylY+S0&d*-<342LPK?M#O+k!P_0B=yJSMwM@5& zId7Z8ig=fpam+75+%#rm`<+jF$;@DbH3mLX%lCpbVL)FgQH#2m#V?9b&X&_|6zbP{ zx)P#qyOo}}rO_iTm=>~}gchGI-mOq7d!HA^7P#D#i0=)^tL&i5s?hdt8e2FbDVyV* z6IMoXbS{@@q0Dheg&11zt3QFvwithLsU&wIc30ke&Y*Z@jImao5%MTm}MkAG(e zoA7Sn;s}CE{u;_O%G`&dD;T=AO69OkwCZq0r**#m`KOX_bBMEL%!PR! z^8l3^^@f0@V@FZfV8%}zqGyl4|MA)HUc7w;F536yO;rYq7Qeu!J_>>*dy;wo!zzV< zA>h5E+YMmjw&jk4h!7DQxG0wO^%?`xuHjyUO2BX;l;4H9{rtIxRC#xdIdXivL?9E) z7f`ydkTi-I)Z7({Af55VnI$r>N~2b%;B!ZlNs5O+V&IvTaU_SLF`Y#TeGt5S@Z~YlZ;kB=EO;~u z{4P_J2AlSPftrPT!GUT)E_0;2Id>~)=~`s?6d4a>eS=`;iYJ?2d7mOQ8Fe9XijGJ$ zFsf%1#=~1uM9z?48wjz85woHl&l(bv65Q5&0_?^iK%lwNR%g)SNp2C?Wd#E{=MUQT zpv2C(^Cw&x{75r`isce7pRgq?wZ)$1vGEJ^$Ob8&rYHY`7B*`tcUCKVDMfcND%0#v zzFazl*3=osL;YqrRVlPDmhX))y_#QwQE0{)#9;}-4|Rh_FGse#FRX};Xzbe_dJfVq=kgb(Wea9AW(kwgzL-$RB90%hX% zrY%W?{bu7$l7Sw?oSo}|G6+L+{fnYSa58coFMgBU^8O)+HyOrhb~&q^`gub(Oht-2QGOpCIi(-H4VvN~=P!N9#)* zPUg%0bQ|D@6`+27k>70X%0@E;BKhxUKG8-9U zdtij$YN9}-amjP_DE$Op#zp;MkYsJ~GCxGIDPzi9qWx{cW@=W4#M^I~^cH&Hqasw~bZW>XEIB?rF*UY8b6!zixN43ZdA^Y8LKQsIwp zs-+XFU(YzJl=lk29Wm+aIp}|49*hEFUWH*kH~;vCeuqy0H>%O;%qM(_gna(#uD$}k zfcF$BlM>+&MDm&`AMY8BXuG4?oCs3BQ$Rw@lU^pD=Ld*Tl)ST;#U3R*{?FN`gZpsgoLj=y^#2S`Q+@-3jvm~;avYkzfU+wV#*F7-! z*o>%-ob(Gv17Xugq^4qTF2*#i_SkdODh1pnL6#FaD;Ur0s!x#er|RP1Z#OIRb;2*K zqe+~o*iXpCr`0!(-X_L`h*^n6hr`wru_8-=@YjK=-wa#T;b8Hp^Y&%bN_oHe$sJo5 zrXPfPqr|rfHtr%jBV}uxMBF6iB(eD>_{|Ybk@^20O!R?q9Z*KAEp!_L)z?*QhBwc^ z$;s_*4p%_C1Z~^nnS-6$N3I?m(s8V-@*NR546wH&${6GDaTTNEWL0*FfG936>;2s$ z!lU2TAkU(FC5z1pEWA}TGR{xakOyqgE%wH4jfTTI$}gEXy}*M!{`c`}!-nQ_WrdxO zL^SMq+Je^b96AF#R*ZsbnybE z8O&Aa&{BJmS)NK!o|Ok9Z5_ulAsR{~@CjEMYz`xc9PQMl(cX|h1MX=wwBR8_>e1M2 zC+#>x)XIIN1SRoeTt!d`dDu1w@;BYZIaK|{Rl4kME8+)24GHys-?*_|?FE|DqoIs# zS5^q-ja!{J%y%7AG(A(@pq3jmW4}@=8s%?;Lq%W4JRdkL=JL6-R81sKt!5~;_x1p3 z@mO%hwQYqD0F1k4yVn5fP_!UX<(EM=O+LWlGU~KQ(^0ut=u+cemZQI&_0KIazUp^c z3fquqj~5GO?z{APmlnl<6(=(fL+SnYmKjmu!%6WLHde?Rz;6wU3NilfI8KQ?sKG@I&%AIY|*@Orx z(?;7|unNm6znZx5atUnW4u3vF*Wb*>RDGK5r#&lQ`^pUKON>=%mBC^q8@KclbI3kO z6x6v8`vr&ccwet7fsZl^k*^D}=0btfNI+nY?D1v2yFAp`7EqpsF7KM;{a8Cl$X&G$ z4B>x@HsKAn|2bo3`O*xGT&0EsHY@HqT^f+3b&o*@Uc~xNu@Jy@u zrg)H?P|x?2Ji**`G20IxOzJ+8`hS{~?^OR}?iSL$p%Dtze$e}~J=h0Xd27V#3NrLG z2b=AYuEj<@!ox2B5n0_JLU3m>c8FO(B}(mZYwkb6F>h|?>iZ*&sN&>F)I*=vn|$@C z*@Y%_%)VDQnEmf0v1Q1Cd;{uoi}+7@F~!9hcG7w=co)NW&WhhED~oIfzt}$b zM7hEzIYNNE^TVxfJllqWyMx|%gzPSne!K!H7^Wa(_FaTm-Ps&mf@s2Sv?#OeDH~`C zA7BFzn=aUu6rFtW5AGZPZrZNH7zEGd7Yj9(;h^Ry#e&o!^JYjU#uyjj)37Q#5 z%(&r%*a;x@XuGRFz=aB+G!&T%m8l^BK*=;~IXAl4*GijdgF&@L#5h| zh2Sa;G!G?79gSX>|N=c^(BEwB-6iJ@aJ&ZH)11VS2u?Okr>$5aMJN2Xm1kM zV*Zc+^iThF_*oswe;3;C|5Peqq`q;zn)>!;v#PRV<;>fBs_i$MBW9%4bD|IRSZQ{K z{U-O-JoN$(O18OQ&yw*WV@Jq%b)P$kHGf5~3bilX#nGS%_pLH&sW;b*WA+FiwrM_b zL$#w{3p&tr+Ms~_z{SKdh$0>giGI_MDu<0aI6=BPcd8`7q82QVI4n{)|FK7hyvQ1r z4Z!=Df?HU*|$n~vd#05)+-F6iA9zmRI|+huu4nGg#2XGJgpvU*nQ>;27`@@{!! zkvUP8o7*%YJH{72N=)6r@@vywCy1TX7D!@F83!Heu)C~WFo-^IlD0o({dQB;Ez2Jc z-#s|fRC>bMn3@7#ex{ux|H)yev?a{VC?$+iFxr z_J|iF(Q0N1n$t1w;kR>{4g#x#TJ)Hc_xRJdrG?SJfOcEmMy{QI+3p}ZXMg99Y-(l` zw$talPV>kHkybvGTpX@&+)xap#SKurWti~VQO)HBODY!o@_m(SqKjh;+xZlNV3-;X zzvN@hy@YS-^`%IY4%YLql@z0yPEuQFWhDCf^IDJ6uowL{?)%vt52K9&e+Q2S5>rIo zXQTi)4^Ye#;XzsNns5Ys8=-w(04GiFj_J_{Cj3BljMS)*e#3QqfQUYdHW+9tBaCch zQK$|+1B^)3aPpyS6pjtG8Bx6NzTrR|44{$qVI*Q0@MBXTle=dk;^vn|l?7tyD?N-> z#1lD>&Lu&j(8IrlJOz6Z_3uM~Zt!@Qv680X|M&m=zYh5WQkazrOlJm4^~UZovItrT z5prH*_UlDX#j@*m)+a*fsBogzrI=lVZSg>#ycG3ChwKeoBV>uQ$OmWnb=Pai+M0f! z@TcAbC={oASQ?#vl@R3!ZELFWSp^!CS|Vd~d~LPt(%H3>Afl|o?Q8lLf)UW*Y{rCw zOaa(@$C0n*@R1vAdq;ZUuk+fY zNY}|#y4TJSjwnVXc? z(5u4~Nt}$^8{!LK^r`0!6nsw;vpaJr@3L%&c&J{=jW}m%9&j1=u3y z=C4ux@WBF&u9M$@L{t9b_U5YLU0zJyh*x6Z#erZGQB`5AiUk z-?PHCy`sQ`LNmWAoA>^_auvJZVgh;yTcV$5$qI=)9i}~^q*ykinD^VLV6azscxG8< ztx*d_&@3f#A zDVK-p8S=fK*zSuHXLHzNtivT|h(wA}vI`h7vpyMSnBj1+-cf_bX=~4t1Y5BYj@F~- z#XvpbXS+m2@!6ebd3{nWfwNJn7zB;&+_KSlcWhig!?S3{M;(NcGo3X_tslmJ=Kq-s z;b=vVOhOb%)U9@YiVmp=Bo(3+UOvWZr4xrq$bKaMq_%l1cdHX>2`%d}T5V3G2}nD0 zVGV|f5rF7v|4q?~Usby?8D>xbftWE@vv_-)pL>mv$si;CB?7VxAP5SsFpOA78;xL+ zbi|N8q$^Oi-yAM*zN^MM+)!18t8Y`SO7j8-0vEp{yTlOFgzhL{s&*XwahhvVmw`xT z4Gk?QUZTW`O9Vsu86R8DEY>c_K_zsH^K)$;%)o#;PJUV|c-vL0U?HH6^@tN@n^y*? zVGnIy64NBio*U&<(@fh$LghEhcC)pPFEjjexWEropZJju?pK?wyfs0rr6-A!7I3gH zx`ZlbBOfrUTHJ0Km9P(~a0zoxrb0B|HCxCT2jdy=qMEr~12yLoSdp5>(v(sC0Y3L! z90o8T4mu?=dz;6uO_-Y5tQ<2jn;odZ1Is4pd0EHt0Zpxh7DA((9wl|0Pbv5K{!1yWYy2e)dYl8H;EFC{$#wO~g?gIrI%H&BgRU0795o zMu`r$4AY$xs5JLp8~SrKJZgM~353-2pEmpFkSPa_ziNT{dS3s^PU@9W$mh$=`WpIA zDn!W66{=-2yTeq?G-3I-=>(2dv_w;t+_Fbd=l9a#fWVWKIIAy4s+_7Yen#?qT=(S5 zq5r1mG|@VePp4~p@H7?PwZO}a+2p0##FtWhh1d&(F68IuoQ`R($47{i(L#og*;O3a z@mmnpY|C#+($6q!oSgL=0nN zrOg3p+I%T8!G&$dS6tC+9%UDme}hRv;3l%KEXOW<9vLS<$pFBZXdeVhq78Js+JM2O zc=7ht$mZ*{4JellM0_AWCt<0e2QHRUGRon1#+c{AEbzvXwNTU#^TX9@$3%jNj0&)B$^^@U_fvuQ`6YCc$E+1QC$9AOkp;B6qD1&&Xh(} zfc6c1k5_QQ8u61bxf@-4vqJN*<39F$6F{-uCHg}__BQB8)z}@S`w66YdyAv_v0fck zeCe?7#=EpDH$-ETLo|AJh~|^CP{?!_Tr(2ehs9H==}t4Oqfl+dN$Y`&#Ti8&s3)$p}?$Ef>Pt ztibDv64GrjFWk9)MxMuRoAfk!EAKIWJ@7g~#soQozN5}9i#KC3^0Os4PJWcR3;CPRs-_dt+NdPu5qiA+VFQ%fD zFS+wFtmHe4UMfM|8%4qRF3SKP-u4j1)6@LteH3#xc=yxKKmKU0SomM5iwyp07(Mvs z;n|ScU7d*jeM&(ZHRzx5A$Z@@jtxOO`UK4uK_X1F#QjPZ{7zng4ZeCmn5nnS$F!=`~T(Ark3GCr`@$P}zhka7xogP=C_?A({MnNa> z!T4buCRh+=Jdszl2b>iwQ;r~ z(hZ1rDGAVzj}%2ZU|NqWcB1p808tU=dUWBr)!7AdrS?)*Ts!J3pwm5>tgy4)JM!C3hcI#KON-&rV8h29E&XgR_Yrx(+;_0) zab;}em!XP!3Z&0+mVO>u#&bnbVQjL+!b{{F1lJQKOC@4UL$}naz2CXaQ1K)bu(n}A z@Fgf0A|0Tm;7A{a)OO|8UZYYzXLMW=;sjEod<0cIaC)6uyHIRjP=^&%6^*YkR9WH- zVycoHVBAJJ+e&Kkk2WY7``H(eoW#dl`{g9ZLZDb-$Q^AH0tcClZ8G3o z_EoFb-X|9?L4ev|B>(BL_z&r^V0e7rG=xcNY@?oM(|il0QbKigqAg8IKZJ>dDxU&0 zI$I-xs}ocesR=eQM3gG+L3fO0&UG*02O!kDu7p!Gs~KD0N9h* z&}{+3AL{BB)a8VU--eGboFPoH7xv2}Dz~B;DlKFDAaA7~<;>r4GPo?bF8ZNUJk5+G znRJ%w0^!CF^3=)SiF|BGgHR0Om%%I$b+`CBNkT;7)>Czai}%%*9bM%u)-@I}1Od}E zXe~zLL6{qyJpz@@2705ZUj(Y99Hf+9C+W@Ca^-kfE&lnNLhJm499OUWa{ZKG9VS%VG@6J&C&V_MIHxBD$HmAj_8@L@koH`FUhQ7Hh zsFLi>QCOp%Gch+TOx6(6m$GljT$W)i6AF?N1~96!Ml>V1lm?eKw3umdq@BPS#)*s% z17XdwF%qen)QyqU)gT$`k9NJ+b*20+f?!nox_-&y_-PjfL zRWXn`E>R;eVv#r#{0)%U=)oK5S5g$^0)h^?Q1*?NhdAv*DBjf9H{<F*;5)zbY|WdnRSCS(&l8i53G|vn`H~co6fmBtMk8(!*MGVzKmlUxv*e z2TE?L)@xl-UT`;R9=l7$Nu{p?E0xFx zi>4~T0qn`M&afJ=3<531g00u)I)x+*YHsG37YZ&e(ft&4HZj5}525&ROq;t+j|oow z*O)eXvAJDaLz{aQ;!f9_u^BdHpl&GZE@GtOM6*&{Ur@U4{qBH16pC1+w7#k0sq5GX z6EMx&)hnxjIE)pL+`xQI)I-e-hWP6@Eep|8U|EhO|6*qB%6LPnobb@e4Sz>5D3q8R z3>}T9?2s;I!QElz8Y0|ru2xkn4Fh+cj#jLIPWVUs_HVS^(9Pf4!-F+ZJz0{=J8nxy zCI$cMH#6yt>9bjr$X2`gG{vq*Z0jtc6&dB|EoMJ-wr*&8D9Z(=uhuILa+uByb8Ul#ZO&Z+?Aj{WqHB+_uoj zBO8d~5QZ=kuQ_>8Rd7<(g0iX>uA*W~Z%@Cyj9$*@qD{k3ta|=VVG>T}sGjbl`ef;epBvs%Z@%S|wg;AyEyyzoe z5zHAnii+R<7Jz`)O`B@t>WW+l?%hZZi5^*RQ0T-E1^(V~U5L{Z>!&A#E6_E*H?PlZ z1@QZ>;+c-2#P2SvdMTGrVDAg8SFtGqat8VrNBJ=-#=(*dJPP0KsxU*3gOx?FQ;I|s za*CP$;xm#Urx2w}`}MoWzb^=mUYmQDs1L8E0MQr&x2Irc#bsHc6SN1w+1=!vj=w$Wra(QTgzsej$73;G10MkUHJc_~b0yRF(ba0{u|EPIZ`%vcmC)Vlj1nU%u&O{O`xFzin&~Z!AW@y7A zamylrk-3jP1UpSAs6t3l+CYF3QEF_4z}SY2xgkc&vu3)Q!r-W?No(7-pQCNBK6Be5 z@Xgj|(rEa8u(zMmCuJ>2ao@>NKS#yIT_XoFq=*3wpZtdS2j+GmM z|LEYXnC7G9I4P3@MQRd(dX%30a3GtW^->Aa@eFGp7Zxo9(`a;C*tQLcU>8@t4&{ND7Yto|ouD_r9oMR73o`xmkBM>PuNc zS83_@KjpwFNX@s|&=iBB>eU4Z(=3`xqtNW?jZ_<>R?Z?oLN{3SGXb!~?hLc7h7C-? z4fn-t9f^r0lV!RS9>}3IYStVnL(C6|ehu=`R^`sj5w0F`#2Za?SzYZ>i6d*Axg4;= zxTNYF3CScI#u?2ZV>U9;n*nBPdMt1w0BzJg&>EUGnKSdU0K#5kNIdX&T7Cpajet3Y znK%zjLRxj_v4}jM=l~IY3c*7Z0XzR;ma29-$YZ+!$c!VdGV@-mCoM;P z;<|nt*y0qh=|7}Sr+!cupQJeC57=mVrzf8WDw74;Ng+bvzB#&beF9ToMC*KiGwT$s znMRs}22d1uBv=bUc8J)XfPr2CpTR$mihs2K{_kg?;3q$y<*Gx;(AE0cSDEGcq2RC;Y zyPD!SmW^_}p881p0(162RYsyi_-gTewOWxw`)Tm(V@X;KxJP{*$%(Z^&AG z50hCSL6T*td-h0-n*9XEO&0<#DCi*SW7D7k#n;I=(9-lu4U^|^*iSU?q&Xvc-;Yj< zCyBvH$@oCWFZ!;SLcf!6HG&l9te)e!U zDmK8)aa$u+IwF}1puryf?Wcokc_3#1AAv18aH-HmCRiw(R^%OCc;UuAaDoIJp7*+U&IO@`7>10I83{1yno`>0u5(S!xn)*;es#6BkUWbBS~4V zCD&Rf$)zIjeJOz`&RbVqZ>rS3FKne0lKS}=&}MQaR+Dek|~PLX&D z!$fG(WZEddIx(sQas`A$CKaIj%mA8AXVZn6{mHo{jku&hbld0ZBlu$K{qk-Sami`h zswJ~djMYUDlcKdK>MtJ-|9^W|x7@~!rT2QP_8mIXh1oVqo3;~sWiz!NZ)PV|agvQQ zlM9uWYLhLgWBzE^q-E`7ruHHB3HM1B=iu)EI6yZk*@;u#7m3C0Mgs@H!TJ3T8}Az& z>us`xN>sjeyUy-{Of^`8yo~&_V`>bRyIPv!0D{yRv;Og3o~Ksq&&?}V?Lm0_lLUSF zW8j8UaLZV69?DJ3YHfD&EAUBpL+Vwa;i7JfPFz}pnzvg{ZR z$5EC&%(6#W_BhKv&$2JF>`BI2dw$6V;+(S@lS{&d@(ZH1~B*$_{1IKdwe{h=81B?_eq%UO*J!d~9XXePv=Tk`6UgqQ#7iTcG*|q+uK?vS%TaM2d66Uav#JCJY3GKjrjKr24y(4 zA2|7--#AVkFX@Ht-4l~CNW`^$l$;G`kl%jy<5#_xLP5)qXi);PwwDy>A&kMmC^70T zcnvwQcQhpZ8VfaxwKNrnJ|yGHWDS5wX7R0tYMyO-53&n@$(6P(Y;IZBLZQ-;Z#p1j zTGZ3R!T-Cb`|1b9Y82kHt@u>gF#iu|5Gjk6bbG<1Q81g7#9jEmrI^(FSDLfNcTQcq9^#ZTaDR}dvy2ZfqxS{6iM>Y$K%E4vj$X{v)FhEXC6 zWwCw>=@Qn=u0)NC)tor_wVbM-}X zuD)RAO41vrk<67Nin&R`*dg_d(RuQS5%yK&7e&h5h`Kbhew6<*H^vwhx3D zejpt!_hJ&qaM!r{z6_M7@z_DrO9&4i9<&LC@5|Wsy_!pTA4Aj~pM-{XN`jHO%O*s@ z9uqbq9vHLGTOm&qpKd$%qWfJ)`hFqlw-#9}ghld72;78oHnAF)v?WWFj9v<9ExZ@> zhu7o$*Y588RvP#+HKGSjxOb_A4~ypy+*fJrL?UtC(mBg z2L82C_E2crkF(FSFR~}uFWJ-VU$c?;d6bPFinp@SW1*~O$PSt1Cw*?d(~Y>oc5uI0 zKio|malSK7dE}mpP6owJ&ivf!-L9ld5BgwMmrslDwi_{;qu9b9Jwor`ioQMJGZ?&9|-6kX0SbNXJa;3g$ zt;iwnU&D+@AM=CXw|A2*5Y}sUY-(`6v|@k&)Xk3(@HDqCAIJF_x(ovo2H@33j21h% z4PO|gcC`U5bw6Y(+Cn#@F zT?LKR7|{lWRI-cJT%kapIi-I=hcnScP(gl}{aLULd{VS>AB;T2LIYUl62qiWKA6JI zZ$Pe3@u8dpc)$vQUPTPU;fA3&n7XR_Ssrk97{yW!x**#X*O4uTwmR|P`C zsBB|X+)_)!=m~LrkbVQUcwJ?{3*S``+zHB0nV;<|GHLS}v(j1+><5QJ1Tk2UuM*?9TB1p1AY*oIly86~ ztfFHf6c#F40(`<6Dz$&NRPf;MH#pmO>g?s}CpTpPXId62^c%#g6^5~@(-YMNdi?&7lFX^kIG?Df!aJLgY3~V$#5itbb2stTuq||6jkaEm z-$cDB$3l3MtUWGmagT~;zfyIe5QILR29c~L6h)P%zASx+Kn0XKS_;T@boqNAtl+a} z0iUo#3G5daUY#p6lO#aeyl?9FusKgRv-Iz3?T=afZ9CFAWQGqqdbIk}2lE#=J_8wW z--|nhOvm#Utu@sf3OI;e7p-wQ+PfB7xPtk7;tnrKQv%`XB^Iet{|S1Vkca!(sU_j5 zv399kFRn#ZXw@q>tBeXjUdPHH=q}dAR%Q)z&5?*xHLE($tk`hsrA9nT?b2HTBGE_^ zy-IVrU6CVF?BsT9i6z;EOkyMip0CcUsVWIsV8CV^wkY%{!Vk(OfpMDP^Id{pG1`O| z_khZkd^L4sC@4FuNiWWJlykR>#kEP9P^KiI!=g>H$+_CdD69Ndi?K1wCgQ$TQ5io& zv0@nx?t~r+He)%K1ncbpKz?HYAO{BruVp&1rju9MSK z_|)ksK`T%NoNRjfGnmR^I|qF1r5Kkus=y(0QM{{G>)dVafoMu14M3`MWHQx&q*=bqVHbG2DK$ha{cu%Wt6I~7LHUW~7Q|#*$%icZHygz>y)>O2O<|P6iJqQ(gMnq z?}fOo)_sjmA_qMcr&(bynlyhG^hD5;=9bLXg^(>37#7%_X?xAfeF!?5Yt?t0i9qa;9x*Ar(CNNWIWnY>RM z<>WvK^RW}Khul8D60%2Me9J}lfCyDg)b?OKEOZe#wj{~PqA0=y#{@_=iq2BCdBFV_Q2l(S+!e^X4~g72C(rkA8%i)?H2)dn1U z3+Nt(MI+`To^EGGtEfq3Cx-u!A(tYREa`RdQ`I z007SAt*Ws$U0q&Nm5FxD-gq2N_@*>Xl9sT?W-B4FU9 zSWK7e${uPkYE>~Wt3nJ|NcSj6!pAV++>A`bl88u)5vW$?t?XwXZ{?xqihDhvn;Lhwpl{`;vXA7d}PmTW7i^a((PR zcQUMM+&6gxe2|XWTbZ!3TV9Rj)g*`}FxE>qmVo4{tTwPin#CxY%hlpRd1nGrFv=bC z!1{Z6IlrD1GbogUvB|}ywB70|C;eyzzx}t5IMkL>sZw!V^6!a>StU!ijvv!tlu#|j zEgtrv;|C{9qNzSCnHkFvqK}yp*cGWg37r(5igEoH!Gx26_!yq~nY>F_FLa(oCqYyV~D9lgBT_o(jaxqxK)! z%^~7u^#%{~P&W<+;4d1SiXCQ{nG$9?Y#^zf3>8ZXmMtMB)E6WW8|%^@uyuHmyXhtI z@FYikw0kLXIorTbh82x5qs=4yKYa3dFgh8MkPsoury`ySvt@#nL3fxI%R69ZpF?F! zef*u5%W_>PxdP2ZHb9us9s+e{m5DxMi9Ii-m7>xB_0BK1rl5?VYrP7qX%+;(Ko^|9 zlJJ7n5`lKs3f4g=>Z|Sfg+X^(84ba$!l#_^V0>P|;I(~&beN8gxA6OtXE>S`y5?tb zUGs(RxskKf-%5SCnODzYR!n=(SX&VN3_wTawvZybv^ zU_>M$q1#i)e0-fv&J$+h$gEFk|^WKdcA&60W!_NA2sfG3m zuZ4Sxrm92wJq)&lV1n8$?lNS?3BktVHeh521JfhXyu=Xuf(bqYPHmg#;TUuJoXB& z@9>^;|9O9c>zwQQ{X+xEIJ6rlCOz92_$(rL=vw+=EfY9Hp`741gpVy>lUjYLA6m%c z6;@N-fkzX}b6{9_2L0q$N!fW5ng3!(nF@e?`;ns6us$Qwy6rtmma^f!5f{P(n^2n% ziN#UYuxq86>7z2`Q>P3?*yH`5{qXg_K(3m#l;1s`{CzW^r$7>+po&XK5rwp*{ZlB$ zh}k~=4SSYKW;p}sJ#g*Yhd&cMFV{O?oVPp*HY1&apvrRB{OBs7EHJGJ7d=LaP|^pF zw}Kq(?(_`hdXK#2>pXr*_uoMxM_@7=BVG|RZWA-{hMr4$z9n3&(2%O-J-w;p6RdFa zz_QXZOSK&6oAB7%Ir^OVm$uV@mR#w*tS@5R&8wL+dVgKylLAI+($q~-y4iVFuVr%j zvNIcPE@^Djr8lx#3`#F3$pn-3ExxEbijVIou8^^IIH=W8Z1}ZT=k)W)%M>!7%gip+ z?gQ)=vNT91t=lA?G>YscOEtsP2QEhgb_{MuQ+GRKo(1=B6S5p$6|x@Kbt#9jfW8;1 zPO((PyiE_Z<@8j8d^Gy4(u=RW!jaL^U-Lm1jZ^beQ88|bsOOt^iQ1*ozaK}4-lARf zx}1`HKe@EjcQo#x?;m>!7z!I4*Y@E&WBT9|aREoMnhdL1f7ZN0O%7htQCdPxF@?_U zoryHhq}$%%NDJvk!dzIFnfnDnbG{2lQ5bxf!9`qjX^q~BD$f=5o^?0QN2Mriq?PZ# zk*E(B0cs4}*NU2JN0C~o>%siJnp#0KCL9y~`I02PPb1NRU@$QZPumI?b(>PzYGYj| zq`+2LyY!K6fTl1Xce`*Q$RztE6Rz5 z(T+~C&*bf!+-De?6 z*D|GEX*~7eM+UxHXrT3k!KbdjF5McHl7&gGPdZp+0|m0;j7anclG7UJ92cWidFG+3 zq4Fw;1hVnGkIual)kk$3$B^w`Suuw5-CjZ?6u%n;EMKWYn8V2Cefs>9hw)o;Fk2e+ z+_6o*FdNGERuZmiiufw zoc2|*yH;}wu2bu7q2=Zz3Y?Out&YK{d)uvuFG!2wK9d;F$ z7l#17eZPQoNv9^)f#FS10m8<;8B-?b&giG(?<+s z4usf5eX{flSSi3g@k}^>G287O#a{u~_pjWs8}H#IqOfwx_a3v(6jrIRDW3H1vw3b{ zFR9WXH)|OCvMp@UuzTa=PV(1(B5{@;5!K33I)^k;JbammGmQ^?RN}1#4ts`r^izgW zFUCvlx#L&iFhJ~FLN(>fh%fQ$o&1mT_mZx~c%FOx5+2YbcxK^;zy)c}av}t4$uyRd zz-+PBYOSs#7*)9~boZMlaiBvbkI4(uu8O5VIt*m!oLF}S`e%~Xwsr+@-v}aL!8}o_ z|7Uabm4pZH9G#TAd6d0u5=%E!^yTZ*o-l7dCryjVUTjp8=qY5+g=Qw~Yv0G=^#5|}`j6iFV|2|*hPoccZ zWq+?E&GlIca_GCVjC;)yO>nBQ$#F+6KOG1_^z{Kj5S{X4Z6r8fk(C-rqL7Tlvqv4&f5=(E0VoDx~@)@N;Iyd&FrcLCpN2 zLhF@lxzSvu%oJ%|oq)v8_8HdfFY_MhwsW`x!p!8K78H+@$q;4J3!z9=jWyniQ0XmdCTvygVaZPb% z%IWvR0t`ruRDO|{hrEs9Wo|CBOgETD?Yc@$m%w-$le;{M3$qx$x`Sx+y}4K>oQYXa zy8t&qYtN76uwJit*%gi)*op{KTUomTXjE8u%v7>dO{?qPkv%> zp_xT93m@DIbvAilm@X;@nP@tgqk+t7`@XvhlxaX8gBQ02_RqH6&Rx1lv|R-9sLyUp zVO(ay*vB1tDpGhl1n8|wZuKWPb`K29o3xc@4zF!7OoGaUL|>vvy3}Vh^~ijDH$%K) zR-1Cco+WQ5EUP_4O0MhLMZ(c?O)Sp8F>@mL5+T4jWB&1b5;h`nYHGM?bmDqPXeWmKAa`Bx{Lz^8bBh9ot$IWBn=k7N4Fu}^QOk0}ZiW*rCbFvWAG z`C4D;GI(-|w@?}Xa5bGI~)Fdrn;g}3@I-8lVOoSXy9 zKoOxrWl_(RWVL7&gV^4MUz^NvG)7^W%($(KUvbvC&d-XtC&;QllhSMfjVV( zzq=K$=HVRNt3aDtMJ|QVMWTqrMNKoVzIB>k>&=FeP{Gs7GaM7Qp%ckvWl8PKR0Y@- z0`G=<`QXt`JDm+n5Zq}j5MkICa>w5o_yHP-_3{Oi-u6i$`>MQ=Z zd4lNW`VT$_+56Y2l+MBCBp{BHT|b*%p`yPv;_4@DRykuhyqn>oleR0YNxaZk6fqZ` zLGL5Q1Lj`a^Iu05f2qk0nZQpf|MLAa+{GPtwt0wVhMg-f$$=jVeaai*1{kI>@7ihm zTD{YDz{MiqI9@Uorf3=Yqe>9Ni&%mA3A9gYfGNRh!Lf@1zV$9n*ZAFF*T}?RcL}xd{uA(07gS?0jPrEVF@vO;CJt_aC%lqG-Q4G94 zuUiq*`kym$C>fyMSdM?U?&D_S9s?Nh3B_>4f4568#0?xiV9$7E%Z`Z*`GFW>?uwML z0KlFRrUy74fDRwF#2qgT4$bo#|BRzsW|u|5*<-;0{p@FHQMHA@nkzgZiY*pf7pf(W zLh5=63j%hTV8r!gc0ZdBdI912C>1ESZ%aLlhcf0HFS*fc``E@)9;GWWi{eVPh)R3J zU-Wy<&rc?i)oL>=?xrTlew_g6dvSoNbqJ?4#xIGKK>gSWKjx5EO2}0;qfTr5_Fae{ z2B!6%F0;gKzw4<~2RwnIPDB=z6;f9P2RpG@Ky;^VZ1 zS^5mQ5yizcOHR3ysGz$(f4kGW-={T~ArzBh1MIxjy|5+rV^cTZ%s4X5qP7dBI@m9Gi5`Dv@BFX%;0Ft*l*@gMe8v#tcYyru1F0TQ$NvQ zPKvhDtO!;-2s=Z@)wCeAyet2lQPH@FZvOP?tcSGWa}`uQ5NG3cRdZ3Gj$L|n3iCO# zx&j&c`Aw<5iPX3ZXrF!=$hh%~*g4UhbFxoXc_qG)8b?}aNU&FUo(T;~h9?kRT7f4r zUgP?mo)yVOh_vB)7gicmr|@Jda&Yt=30IYq7DibFY6Fz)tP~GOFtaKd2GCPERQ^rV zkFOq^j(fK`&-5}U(?;{A_c6f4ypVDFsR76yi3g(BkcQx2hsbQt+ZZAJs&9{z-A<^k zU-q|&*xHtx3^sn8)lw!dpe>PeE6&}r2FJjKdHXbZeNrs6iklv%1y(r!7zkuWE_6bh z9-m-u&URpo9jnqE)7INAcKi%aDi2imW;URu?bKMld5g9=jXsi|5HaG=0beF zsz5HcRtT Date: Sun, 6 Sep 2026 18:08:12 -0500 Subject: [PATCH 3/5] Pin instrumented cache builds to verified maintenance and attribution source Select the tested SparkCache package with same-pass survivor reconciliation, request-attribution accounting, and optional deletion pacing. Preserve immutable published image receipts and package the continuation runtime with the startup gate. Validation: 3,089 offline tests passed with 85 documented skips; Ruff, documentation links, and local context preparation passed. Image build and serving validation remain pending. --- runtime/glm53-spark-mtp3-mesh/performance/Dockerfile | 2 +- runtime/glm53-spark-mtp3-mesh/performance/README.md | 2 +- runtime/glm53-spark-mtp3-mesh/performance/prepare.py | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/runtime/glm53-spark-mtp3-mesh/performance/Dockerfile b/runtime/glm53-spark-mtp3-mesh/performance/Dockerfile index 06fb8ff7..6add0358 100644 --- a/runtime/glm53-spark-mtp3-mesh/performance/Dockerfile +++ b/runtime/glm53-spark-mtp3-mesh/performance/Dockerfile @@ -4,7 +4,7 @@ RUN --mount=type=bind,target=/performance python3 -S -B /performance/install.py ENV SPARK_GDN_PREFILL_CHECKPOINTS=2 SPARK_GDN_CONTINUATION_CHECKPOINTS=1 LABEL org.sparkring.runtime.status="research-only" \ org.sparkring.runtime.profile="glm53-mtp3-cache-checkpoints" \ - org.sparkcache.commit="19873f697c1ebdaf2b11d2013f2411b31f9e0f81" \ + org.sparkcache.commit="b5aca7cd3d3f7e7a14636bf6e5fa1f50a9650168" \ org.sparkcache.cuda-placement-sha256="2657cdd2e54a097c9544e4c79ae62c0646db6db123ff24e4f0c384238c3a1e8d" \ org.sparkring.sircl.manifest-sha256="c0fd5567442b08b908cc193f36d0864e262573c7e5d232509479a823cface742" ENTRYPOINT ["python3", "-S", "-B", "/opt/sparkring/bin/start-performance.py"] diff --git a/runtime/glm53-spark-mtp3-mesh/performance/README.md b/runtime/glm53-spark-mtp3-mesh/performance/README.md index 7973bb98..49de925e 100644 --- a/runtime/glm53-spark-mtp3-mesh/performance/README.md +++ b/runtime/glm53-spark-mtp3-mesh/performance/README.md @@ -7,7 +7,7 @@ The recipe preserves the parent model weights and does not change host fabric. ## Build inputs -Use a clean SparkCache checkout at `19873f697c1ebdaf2b11d2013f2411b31f9e0f81`. +Use a clean SparkCache checkout at `b5aca7cd3d3f7e7a14636bf6e5fa1f50a9650168`. It contains the merged restore/publication improvements, periodic-capture option, and backlog gauges. Periodic full capture defaults off; enabling it trades more writes for shorter history reconstruction. diff --git a/runtime/glm53-spark-mtp3-mesh/performance/prepare.py b/runtime/glm53-spark-mtp3-mesh/performance/prepare.py index 6c6cd38d..5ce47a3d 100644 --- a/runtime/glm53-spark-mtp3-mesh/performance/prepare.py +++ b/runtime/glm53-spark-mtp3-mesh/performance/prepare.py @@ -10,7 +10,7 @@ import tarfile HERE = Path(__file__).resolve().parent -CACHE_COMMIT = "19873f697c1ebdaf2b11d2013f2411b31f9e0f81" +CACHE_COMMIT = "b5aca7cd3d3f7e7a14636bf6e5fa1f50a9650168" BASE_IMAGE = "ghcr.io/fujitsupolycom/sparkring-glm53-sparkcache@sha256:67dc0ae453baaae6831ccec1d259b4ef8b236a8b0dc9f747d901b95c66ec1987" BASE_ID = "sha256:2e41b1e934a85ff7c21b780532db2f0a0e978df081e52f4ae2bf11f8992fb24f" PLACEMENT = "2657cdd2e54a097c9544e4c79ae62c0646db6db123ff24e4f0c384238c3a1e8d" From d18bdf667425ffca0bbc300ddca5121bc541ec64 Mon Sep 17 00:00:00 2001 From: FujitsuPolycom <87842395+FujitsuPolycom@users.noreply.github.com> Date: Sun, 6 Sep 2026 22:06:25 -0500 Subject: [PATCH 4/5] Document fixed-8K continuation serving evidence and allocator regressions Retain raw cold-prefill observations and bounded recurrence/cache evidence for the exact continuation image, including unpaired-control and clock-resolution limits. Execute packaged scheduler/allocator methods for source retention, reserve overlap, publication pins and cleanup, and provide a portable harness with offline readiness/timing coverage. No cache identity or runtime source bytes change. Validation: 512 scoped CPU tests passed, 14 documented optional skips; Ruff passed. The attribution-containing image rebuild remains unqualified. --- .../vllm/prefill_checks/.gitattributes | 1 + .../continuation_serve_checks.py | 240 +++++++ .../prefill_checks/test_prefill_checks.py | 97 +++ .../.gitattributes | 1 + .../README.md | 95 +++ .../continuation-convolution.json | 29 + .../continuation-evidence.json | 94 +++ .../continuation-prefill-128k.json | 62 ++ .../continuation-prefill.json | 309 +++++++++ .../continuation-recurrence.json | 47 ++ .../provenance.json | 28 + .../performance/continuation/README.md | 8 + .../performance/continuation/test_runtime.py | 619 ++++++++++++++++++ 13 files changed, 1630 insertions(+) create mode 100644 performance/harnesses/vllm/prefill_checks/.gitattributes create mode 100644 performance/harnesses/vllm/prefill_checks/continuation_serve_checks.py create mode 100644 performance/harnesses/vllm/prefill_checks/test_prefill_checks.py create mode 100644 performance/records/glm53-flash/continuation-checkpoints-20260906/.gitattributes create mode 100644 performance/records/glm53-flash/continuation-checkpoints-20260906/README.md create mode 100644 performance/records/glm53-flash/continuation-checkpoints-20260906/continuation-convolution.json create mode 100644 performance/records/glm53-flash/continuation-checkpoints-20260906/continuation-evidence.json create mode 100644 performance/records/glm53-flash/continuation-checkpoints-20260906/continuation-prefill-128k.json create mode 100644 performance/records/glm53-flash/continuation-checkpoints-20260906/continuation-prefill.json create mode 100644 performance/records/glm53-flash/continuation-checkpoints-20260906/continuation-recurrence.json create mode 100644 performance/records/glm53-flash/continuation-checkpoints-20260906/provenance.json create mode 100644 runtime/glm53-spark-mtp3-mesh/performance/continuation/test_runtime.py diff --git a/performance/harnesses/vllm/prefill_checks/.gitattributes b/performance/harnesses/vllm/prefill_checks/.gitattributes new file mode 100644 index 00000000..d9bd16b0 --- /dev/null +++ b/performance/harnesses/vllm/prefill_checks/.gitattributes @@ -0,0 +1 @@ +*.py text eol=lf diff --git a/performance/harnesses/vllm/prefill_checks/continuation_serve_checks.py b/performance/harnesses/vllm/prefill_checks/continuation_serve_checks.py new file mode 100644 index 00000000..39921df9 --- /dev/null +++ b/performance/harnesses/vllm/prefill_checks/continuation_serve_checks.py @@ -0,0 +1,240 @@ +"""Bounded semantic and cold-prefill checks, gated on completed API warmup.""" + +import argparse +import os +import json +from pathlib import Path +import re +import subprocess +import time +import urllib.request +import uuid + +BASE = os.environ.get("BENCH_API_BASE", "http://127.0.0.1:8000") +MODEL = os.environ.get("BENCH_MODEL", "glm-5.3-flash-spark") +CONTAINER = os.environ.get("BENCH_RANK0_CONTAINER", "sparkring-model-r0") +if not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9_.-]*", CONTAINER): + raise ValueError("Unexpected container name") + + +def get(path): + with urllib.request.urlopen(BASE + path, timeout=10) as r: + return r.read().decode() + + +def post(path, body): + req = urllib.request.Request( + BASE + path, + data=json.dumps(body).encode(), + headers={"Content-Type": "application/json"}, + ) + with urllib.request.urlopen(req, timeout=180) as r: + return json.load(r) + + +def ready(): + status = subprocess.check_output( + [ + "ssh", + os.environ.get("BENCH_RANK0_SSH", "rank0"), + "docker inspect --format '{{.State.Health.Status}}' " + CONTAINER, + ], + text=True, + ).strip() + if status != "healthy": + return False + metrics = get("/metrics") + values = { + key: [ + float(line.rsplit(" ", 1)[-1]) + for line in metrics.splitlines() + if line.startswith("vllm:" + key + "{") + or line.startswith("vllm:" + key + " ") + ] + for key in ("num_requests_running", "num_requests_waiting") + } + return all(rows and sum(rows) == 0 for rows in values.values()) + + +def calibrate(tokens, fact): + prefix = ( + "Test " + + uuid.uuid4().hex + + ". The project code is " + + fact + + ". Remember it.\n" + ) + suffix = "\nWhat is the project code? Reply with just the code." + words = tokens + for _ in range(12): + text = ( + prefix + + " ".join( + (["alpha", "beta", "gamma", "delta"] * ((words + 3) // 4))[:words] + ) + + suffix + ) + msg = [{"role": "user", "content": text}] + count = post( + "/tokenize", + {"model": MODEL, "messages": msg, "add_generation_prompt": True}, + )["count"] + if count == tokens: + return msg + words += tokens - count + raise ValueError("Exact prompt calibration failed") + + +def semantic(tokens, fact): + messages = calibrate(tokens, fact) + rows = [] + for phase in ("fresh", "repeated", "extended"): + msg = ( + messages + if phase != "extended" + else [ + { + "role": "user", + "content": messages[0]["content"] + + "\nFinal instruction: give exactly the project code.", + } + ] + ) + started = time.monotonic() + response = post( + "/v1/chat/completions", + { + "model": MODEL, + "messages": msg, + "max_tokens": 384, + "temperature": 0, + "top_p": 1, + }, + ) + choice = response["choices"][0] + answer = (choice["message"].get("content") or "").strip() + usage = response["usage"] + cached = usage.get("prompt_tokens_details", {}).get("cached_tokens", 0) + row = { + "tokens": tokens, + "phase": phase, + "seconds": time.monotonic() - started, + "response": response, + "answer_ok": answer == fact and choice["finish_reason"] == "stop", + "cache_ok": cached == 0 if phase == "fresh" else cached > 0, + } + row["pass"] = row["answer_ok"] and row["cache_ok"] + if phase == "fresh": + row["pass"] = row["pass"] and usage["prompt_tokens"] == tokens + rows.append(row) + print( + json.dumps( + {k: v for k, v in row.items() if k != "response"} + | {"answer": answer, "cached_tokens": cached} + ), + flush=True, + ) + yield row + if not row["pass"]: + raise RuntimeError("Semantic/cache check failed; inspect response") + + +def prefill(tokens): + msg = calibrate(tokens, "STONE-7482") + body = { + "model": MODEL, + "messages": msg, + "max_tokens": 1, + "temperature": 0, + "stream": True, + "stream_options": {"include_usage": True}, + } + request = urllib.request.Request( + BASE + "/v1/chat/completions", + data=json.dumps(body).encode(), + headers={"Content-Type": "application/json"}, + ) + started = time.monotonic() + first = None + usage = None + with urllib.request.urlopen(request, timeout=180) as response: + for line in response: + if not line.startswith(b"data: "): + continue + raw = line[6:].strip() + if raw == b"[DONE]": + break + chunk = json.loads(raw) + if chunk.get("usage"): + usage = chunk["usage"] + for choice in chunk.get("choices", []): + delta = choice.get("delta", {}) + if first is None and any( + delta.get(k) for k in ("content", "reasoning", "reasoning_content") + ): + first = time.monotonic() - started + if first is None or usage is None: + raise RuntimeError("No streamed token or final usage") + assert usage["prompt_tokens"] == tokens + assert usage.get("prompt_tokens_details", {}).get("cached_tokens", 0) == 0 + return { + "tokens": tokens, + "ttft_seconds": first, + "tokens_per_second": tokens / first, + "usage": usage, + } + + +def main(): + p = argparse.ArgumentParser() + p.add_argument("phase", choices=["semantic", "prefill"]) + p.add_argument("--output", type=Path, required=True) + p.add_argument("--sizes", default="16384,10240,32768,65536,8192") + p.add_argument("--samples", type=int, default=3) + args = p.parse_args() + assert not args.output.exists() + deadline = time.monotonic() + 1200 + announced = False + while not ready(): + if time.monotonic() > deadline: + raise TimeoutError("Warmup/idle readiness deadline") + if not announced: + print( + "Waiting for completed warmup and zero running/waiting requests.", + flush=True, + ) + announced = True + time.sleep(5) + print("Warmup complete; server idle. Starting controlled checks.", flush=True) + rows = [] + + def save(): + args.output.write_text( + json.dumps({"phase": args.phase, "rows": rows}, indent=2), encoding="utf-8" + ) + + sizes = list(map(int, args.sizes.split(","))) + if args.phase == "semantic": + for i, tokens in enumerate(sizes): + for row in semantic(tokens, "RIVER-" + str(5938 + i)): + rows.append(row) + save() + else: + for phase in ("warm", "measured"): + for sample in range(1 if phase == "warm" else args.samples): + for tokens in sizes: + if not ready(): + raise RuntimeError( + "Concurrent activity detected before timing sample" + ) + row = prefill(tokens) | {"phase": phase, "sample": sample} + rows.append(row) + save() + print( + json.dumps({k: v for k, v in row.items() if k != "usage"}), + flush=True, + ) + + +if __name__ == "__main__": + main() diff --git a/performance/harnesses/vllm/prefill_checks/test_prefill_checks.py b/performance/harnesses/vllm/prefill_checks/test_prefill_checks.py new file mode 100644 index 00000000..d8f8e1b0 --- /dev/null +++ b/performance/harnesses/vllm/prefill_checks/test_prefill_checks.py @@ -0,0 +1,97 @@ +"""Exercise request timing and readiness gates without contacting a server.""" + +import importlib.util +import io +import json +from pathlib import Path + +import pytest + +HERE = Path(__file__).parent + + +@pytest.fixture( + params=sorted( + path for path in HERE.glob("*_checks.py") if not path.name.startswith("test_") + ) +) +def harness(request): + spec = importlib.util.spec_from_file_location( + "prefill_harness_" + request.param.stem, request.param + ) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +@pytest.mark.parametrize( + "health,running,waiting,expected", + [ + ("healthy", 0, 0, True), + ("starting", 0, 0, False), + ("healthy", 1, 0, False), + ("healthy", 0, 1, False), + ], +) +def test_readiness_requires_healthy_idle_service( + harness, monkeypatch, health, running, waiting, expected +): + monkeypatch.setattr(harness.subprocess, "check_output", lambda *a, **kw: health) + monkeypatch.setattr( + harness, + "get", + lambda path: ( + f'vllm:num_requests_running{{model="m"}} {running}\n' + f'vllm:num_requests_waiting{{model="m"}} {waiting}\n' + ), + ) + assert harness.ready() is expected + + +def test_missing_metrics_do_not_count_as_zero(harness, monkeypatch): + monkeypatch.setattr(harness.subprocess, "check_output", lambda *a, **kw: "healthy") + monkeypatch.setattr(harness, "get", lambda path: "") + assert harness.ready() is False + + +@pytest.mark.parametrize( + "tokens,cached,rejected", [(8192, 0, False), (8191, 0, True), (8192, 1, True)] +) +def test_timing_rejects_wrong_prompt_length_or_cached_work( + harness, monkeypatch, tokens, cached, rejected +): + clock = iter([10.0, 10.25]) + clock_name = ( + "perf_counter" + if harness.__name__.endswith("mhc_precise_checks") + else "monotonic" + ) + monkeypatch.setattr(harness.time, clock_name, lambda: next(clock)) + monkeypatch.setattr( + harness, "calibrate", lambda *args: [{"role": "user", "content": "test"}] + ) + chunks = [ + {"choices": [{"delta": {"content": ""}}]}, + {"choices": [{"delta": {"content": "answer"}}]}, + { + "choices": [], + "usage": { + "prompt_tokens": tokens, + "prompt_tokens_details": {"cached_tokens": cached}, + }, + }, + ] + stream = ( + b"".join(b"data: " + json.dumps(chunk).encode() + b"\n" for chunk in chunks) + + b"data: [DONE]\n" + ) + monkeypatch.setattr( + harness.urllib.request, "urlopen", lambda *a, **kw: io.BytesIO(stream) + ) + if rejected: + with pytest.raises(AssertionError): + harness.prefill(8192) + else: + result = harness.prefill(8192) + assert result["ttft_seconds"] == 0.25 + assert result["tokens_per_second"] == 32768 diff --git a/performance/records/glm53-flash/continuation-checkpoints-20260906/.gitattributes b/performance/records/glm53-flash/continuation-checkpoints-20260906/.gitattributes new file mode 100644 index 00000000..6ea0cf98 --- /dev/null +++ b/performance/records/glm53-flash/continuation-checkpoints-20260906/.gitattributes @@ -0,0 +1 @@ +*.json -text whitespace=cr-at-eol diff --git a/performance/records/glm53-flash/continuation-checkpoints-20260906/README.md b/performance/records/glm53-flash/continuation-checkpoints-20260906/README.md new file mode 100644 index 00000000..e1e85407 --- /dev/null +++ b/performance/records/glm53-flash/continuation-checkpoints-20260906/README.md @@ -0,0 +1,95 @@ +# Fixed-8K continuation checkpoint evidence + +Status: **research-only**. The recorded runtime completes cold long prompts +without splitting the final continuation solely to export recurrent checkpoints. +This record describes the exact image below; it does not qualify the rebuilt +source composition that also adds request attribution. + +## Conditions + +Four DGX Spark GB10 GPUs, TP4/DCP4/PP1, native MTP3, 8,192 maximum batched +tokens, 16 sequences, and 24 GiB KV cache per rank. The model is +`local-inference-lab/GLM-5.3-Flash-NVFP4-Spark` at revision +`df116c4fb16b1d37ae43d2cfd624de26ffbc832e`. The image config is +`sha256:489d1975619e9083d14f14bcd1c6cbb4a96c41e3ab3978af048dc3ed2bb452a8`. +Both `SPARK_GDN_PREFILL_CHECKPOINTS=2` and +`SPARK_GDN_CONTINUATION_CHECKPOINTS=1` are enabled. The native virtual mesh, +graph-only decode geometry, linked posting, B12X compute patches, and SparkCache +remain in the composition; periodic full capture is off. + +Requests have concurrency one, temperature zero, one output token, and a unique +prefix. Exact prompt usage and zero cached tokens are checked for every sample. +There is no restarted paired control. Four scheduler/allocator/helper files +changed, with a separate cache root and regenerated ownership/image inventories. +Component probes used a source-identical child before a metadata-only +deterministic rebuild. Full conditions are in [the record](continuation-evidence.json). + +## Measurement + +TTFT spans submission through the first nonempty streamed content or reasoning +delta, including client/network overhead. The historical Windows harness uses +`time.monotonic` backed by GetTickCount64, with about 15.625 ms granularity. +Reported decimal precision is not clock accuracy. Startup/API warmup and idle +gauges passed; one shape-warmup row per 8K–64K size is excluded. Prior semantic +and prefill requests warmed 128K. Each size has three measured samples. + +The table uses prompt tokens divided by median TTFT; min/max describe those +three TTFTs. Raw rows are [8K–64K](continuation-prefill.json) and +[128K](continuation-prefill-128k.json). [Provenance](provenance.json) identifies +the retained historical harness and the portable public copy; those hashes are +not signed execution-time attestations. + +## Result + +| Prompt tokens | Median TTFT | Min–max TTFT | Prompt tok/s | +|---:|---:|---:|---:| +| 8,192 | 2.797 s | 2.782–2.813 s | 2,929 | +| 16,384 | 5.672 s | 5.656–5.672 s | 2,889 | +| 32,768 | 11.328 s | 11.297–11.344 s | 2,893 | +| 65,536 | 22.719 s | 22.718–22.750 s | 2,885 | +| 131,072 | 45.875 s | 45.782–45.875 s | 2,857 | + +Six [recurrence cases](continuation-recurrence.json) and three +[convolution cases](continuation-convolution.json) passed on GPU. The recorded +model checks include 15 regular semantic/cache cases, four concurrent cold +requests, and 25 exact answers across long-context attempts. All four workers +recorded an 8,192→16,384 continuation with checkpoint ends 14,336 and 15,360 +across 34 GDN layers. + +## Conclusion + +The fixed-8K continuation implementation completed these cases at about +2,857–2,929 prompt tokens/s. These candidate-only observations establish no +isolated percentage speedup. The scheduler keeps the 8K chunk ceiling; it +exports intermediate state within an eligible final chunk rather than adding +checkpoint-driven model passes. + +## Limitations + +Three samples and synthetic fact retrieval do not establish general quality, +decode performance, concurrency performance, arbitrary-length correctness, or +other chunk sizes. CPU tests exercise a 6K schedule, but only fixed 8K has this +serving evidence. Historical screenshots used different harnesses/compositions. + +Two 128K cache-reuse expectations missed and recomputed safely, including one +after worker publication. Later repeated/extended checks reused 129,024 tokens. +Scheduler visibility was not traced; this is not attributed as a continuation +regression. The rank-three launch memory gate passed on recheck without a reboot +or threshold change. The attribution-containing rebuild requires its own GPU, +startup, and serving validation. + +## Reproduction + +The [portable harness](../../../harnesses/vllm/prefill_checks/continuation_serve_checks.py) +retains the historical timing and request logic. Set `BENCH_API_BASE`, +`BENCH_MODEL`, `BENCH_RANK0_SSH`, and `BENCH_RANK0_CONTAINER` for an explicitly +authorized idle deployment, then run: + +```bash +python performance/harnesses/vllm/prefill_checks/continuation_serve_checks.py prefill \ + --sizes 8192,16384,32768,65536,131072 --samples 3 --output /path/to/absent-results.json +``` + +This command sends inference requests and reads rank-zero Docker health over +SSH. Establish all-rank startup/health separately. The published harness has +offline gate coverage but has not been rerun on a GPU. diff --git a/performance/records/glm53-flash/continuation-checkpoints-20260906/continuation-convolution.json b/performance/records/glm53-flash/continuation-checkpoints-20260906/continuation-convolution.json new file mode 100644 index 00000000..1b81873f --- /dev/null +++ b/performance/records/glm53-flash/continuation-checkpoints-20260906/continuation-convolution.json @@ -0,0 +1,29 @@ +{ + "cases": [ + { + "tail": 2048, + "checkpoint_offsets": [ + 1024, + 0 + ], + "pass": true + }, + { + "tail": 4096, + "checkpoint_offsets": [ + 2048, + 3072 + ], + "pass": true + }, + { + "tail": 8192, + "checkpoint_offsets": [ + 6144, + 7168 + ], + "pass": true + } + ], + "pass": true +} \ No newline at end of file diff --git a/performance/records/glm53-flash/continuation-checkpoints-20260906/continuation-evidence.json b/performance/records/glm53-flash/continuation-checkpoints-20260906/continuation-evidence.json new file mode 100644 index 00000000..cab803db --- /dev/null +++ b/performance/records/glm53-flash/continuation-checkpoints-20260906/continuation-evidence.json @@ -0,0 +1,94 @@ +{ + "Conditions": { + "hardware": "Four NVIDIA DGX Spark GB10 GPUs, 48 SMs each, virtual mesh over the existing physical ring", + "model": "local-inference-lab/GLM-5.3-Flash-NVFP4-Spark", + "model_revision": "df116c4fb16b1d37ae43d2cfd624de26ffbc832e", + "tensor_parallel": 4, + "decode_context_parallel": 4, + "pipeline_parallel": 1, + "mtp_steps": 3, + "max_num_batched_tokens": 8192, + "max_num_seqs": 16, + "kv_cache_gib_per_rank": 24, + "concurrency": 1, + "max_output_tokens": 1, + "temperature": 0, + "exact_prompt_tokens": true, + "cache_state": "Unique prompt prefix for every timing request; cached_tokens=0 required for each sample; periodic full captures off.", + "retained_features": "Native virtual mesh, graph-only decode geometry, linked posting, existing B12X compute patches, SparkCache; no dynamic pressure-aware chunk sizing.", + "image": "sha256:489d1975619e9083d14f14bcd1c6cbb4a96c41e3ab3978af048dc3ed2bb452a8", + "flag": "SPARK_GDN_CONTINUATION_CHECKPOINTS=1", + "comparison": "Single candidate, no restarted paired control. Scheduler/allocator/helper extension only; four runtime sources plus ownership/inventory changed; fresh cache root. Component probes used source-identical61f5817f child before deterministic metadata-only rebuild." + }, + "Measurement": { + "metric": "Seconds from request submission to first nonempty streamed content/reasoning delta, including client/network overhead; prompt tokens / TTFT.", + "clock": "Windows Python time.monotonic; GetTickCount64, approximately15.625ms granularity. Do not treat reported decimal precision as clock accuracy.", + "warmup": "Startup/API warmup complete and running/waiting gauges zero. One warm row per8/16/32/64K shape excluded;128K already warmed by preceding semantic/full-prefill calls.", + "samples_per_shape": 3, + "raw_files": [ + "continuation-prefill.json", + "continuation-prefill-128k.json" + ], + "aggregation": "Median TTFT, min/max of three measured TTFTs; tokens divided by median TTFT." + }, + "Result": [ + { + "tokens": 8192, + "samples": 3, + "median_ttft_seconds": 2.7969999999913853, + "prefill_tokens_per_second": 2928.8523418038008, + "min_ttft": 2.7819999999919673, + "max_ttft": 2.812999999994645 + }, + { + "tokens": 16384, + "samples": 3, + "median_ttft_seconds": 5.671999999991385, + "prefill_tokens_per_second": 2888.5754583964886, + "min_ttft": 5.6560000000026776, + "max_ttft": 5.672000000005937 + }, + { + "tokens": 32768, + "samples": 3, + "median_ttft_seconds": 11.328000000008615, + "prefill_tokens_per_second": 2892.6553672294385, + "min_ttft": 11.296999999991385, + "max_ttft": 11.343999999997322 + }, + { + "tokens": 65536, + "samples": 3, + "median_ttft_seconds": 22.718999999997322, + "prefill_tokens_per_second": 2884.6340067788074, + "min_ttft": 22.718000000008033, + "max_ttft": 22.75 + }, + { + "tokens": 131072, + "samples": 3, + "median_ttft_seconds": 45.875, + "prefill_tokens_per_second": 2857.1553133514985, + "min_ttft": 45.78199999999197, + "max_ttft": 45.875 + } + ], + "Conclusion": "Fixed8192 continuation completed the stated cases with roughly2857\u20132929 prompt tokens/s at measured8K\u2013128K sizes. No isolated percentage speedup is established by these observations.", + "Limitations": [ + "Three measurements per shape, synthetic fact-retrieval prompts and1 output token; no comprehensive quality, decode, concurrency or arbitrary-length claim.", + "Historical screenshots used different harness/compositions and are not matched controls.", + "Two128K reuse expectations missed and recomputed safely, including one after worker publication; later full repeated/extended checks reused129024 tokens. Scheduler visibility was not traced, so no attribution as a new continuation regression.", + "Rank3 launch memory gate passed on recheck without reboot or threshold change; timestamp-normalized image rebuild recorded separately." + ], + "Correctness": { + "gpu_recurrence_cases": 6, + "gpu_convolution_cases": 3, + "regular_semantic_cache_cases": 15, + "concurrent_cold_requests": 4, + "total_exact_answers_across_long_context_attempts": 25, + "cache_reuse_expectation_misses_128k": 2, + "later_128k_reuse_tokens": 129024, + "activation": "All4 workers recorded8192->16384 with checkpoint ends14336/15360 across34GDNlayers.", + "universal_correctness_claim": false + } +} diff --git a/performance/records/glm53-flash/continuation-checkpoints-20260906/continuation-prefill-128k.json b/performance/records/glm53-flash/continuation-checkpoints-20260906/continuation-prefill-128k.json new file mode 100644 index 00000000..3ecf7993 --- /dev/null +++ b/performance/records/glm53-flash/continuation-checkpoints-20260906/continuation-prefill-128k.json @@ -0,0 +1,62 @@ +{ + "prior_warmup": "completed 128K semantic and repeated full-prefill requests", + "rows": [ + { + "tokens": 131072, + "ttft_seconds": 45.78199999999197, + "tokens_per_second": 2862.959241623848, + "usage": { + "prompt_tokens": 131072, + "total_tokens": 131073, + "completion_tokens": 1, + "prompt_tokens_details": { + "cached_tokens": 0, + "created_cache_tokens": 130048 + }, + "completion_tokens_details": { + "reasoning_tokens": 1 + } + }, + "phase": "measured", + "sample": 0 + }, + { + "tokens": 131072, + "ttft_seconds": 45.875, + "tokens_per_second": 2857.1553133514985, + "usage": { + "prompt_tokens": 131072, + "total_tokens": 131073, + "completion_tokens": 1, + "prompt_tokens_details": { + "cached_tokens": 0, + "created_cache_tokens": 130048 + }, + "completion_tokens_details": { + "reasoning_tokens": 1 + } + }, + "phase": "measured", + "sample": 1 + }, + { + "tokens": 131072, + "ttft_seconds": 45.875, + "tokens_per_second": 2857.1553133514985, + "usage": { + "prompt_tokens": 131072, + "total_tokens": 131073, + "completion_tokens": 1, + "prompt_tokens_details": { + "cached_tokens": 0, + "created_cache_tokens": 130048 + }, + "completion_tokens_details": { + "reasoning_tokens": 1 + } + }, + "phase": "measured", + "sample": 2 + } + ] +} \ No newline at end of file diff --git a/performance/records/glm53-flash/continuation-checkpoints-20260906/continuation-prefill.json b/performance/records/glm53-flash/continuation-checkpoints-20260906/continuation-prefill.json new file mode 100644 index 00000000..930519cf --- /dev/null +++ b/performance/records/glm53-flash/continuation-checkpoints-20260906/continuation-prefill.json @@ -0,0 +1,309 @@ +{ + "phase": "prefill", + "rows": [ + { + "tokens": 8192, + "ttft_seconds": 2.7969999999913853, + "tokens_per_second": 2928.8523418038008, + "usage": { + "prompt_tokens": 8192, + "total_tokens": 8193, + "completion_tokens": 1, + "prompt_tokens_details": { + "cached_tokens": 0, + "created_cache_tokens": 7168 + }, + "completion_tokens_details": { + "reasoning_tokens": 1 + } + }, + "phase": "warm", + "sample": 0 + }, + { + "tokens": 16384, + "ttft_seconds": 5.625, + "tokens_per_second": 2912.711111111111, + "usage": { + "prompt_tokens": 16384, + "total_tokens": 16385, + "completion_tokens": 1, + "prompt_tokens_details": { + "cached_tokens": 0, + "created_cache_tokens": 15360 + }, + "completion_tokens_details": { + "reasoning_tokens": 1 + } + }, + "phase": "warm", + "sample": 0 + }, + { + "tokens": 32768, + "ttft_seconds": 11.297000000005937, + "tokens_per_second": 2900.593077806743, + "usage": { + "prompt_tokens": 32768, + "total_tokens": 32769, + "completion_tokens": 1, + "prompt_tokens_details": { + "cached_tokens": 0, + "created_cache_tokens": 31744 + }, + "completion_tokens_details": { + "reasoning_tokens": 1 + } + }, + "phase": "warm", + "sample": 0 + }, + { + "tokens": 65536, + "ttft_seconds": 22.702999999994063, + "tokens_per_second": 2886.66696031437, + "usage": { + "prompt_tokens": 65536, + "total_tokens": 65537, + "completion_tokens": 1, + "prompt_tokens_details": { + "cached_tokens": 0, + "created_cache_tokens": 64512 + }, + "completion_tokens_details": { + "reasoning_tokens": 1 + } + }, + "phase": "warm", + "sample": 0 + }, + { + "tokens": 8192, + "ttft_seconds": 2.7819999999919673, + "tokens_per_second": 2944.6441409143254, + "usage": { + "prompt_tokens": 8192, + "total_tokens": 8193, + "completion_tokens": 1, + "prompt_tokens_details": { + "cached_tokens": 0, + "created_cache_tokens": 7168 + }, + "completion_tokens_details": { + "reasoning_tokens": 1 + } + }, + "phase": "measured", + "sample": 0 + }, + { + "tokens": 16384, + "ttft_seconds": 5.6560000000026776, + "tokens_per_second": 2896.746817537525, + "usage": { + "prompt_tokens": 16384, + "total_tokens": 16385, + "completion_tokens": 1, + "prompt_tokens_details": { + "cached_tokens": 0, + "created_cache_tokens": 15360 + }, + "completion_tokens_details": { + "reasoning_tokens": 1 + } + }, + "phase": "measured", + "sample": 0 + }, + { + "tokens": 32768, + "ttft_seconds": 11.296999999991385, + "tokens_per_second": 2900.5930778104794, + "usage": { + "prompt_tokens": 32768, + "total_tokens": 32769, + "completion_tokens": 1, + "prompt_tokens_details": { + "cached_tokens": 0, + "created_cache_tokens": 31744 + }, + "completion_tokens_details": { + "reasoning_tokens": 1 + } + }, + "phase": "measured", + "sample": 0 + }, + { + "tokens": 65536, + "ttft_seconds": 22.718999999997322, + "tokens_per_second": 2884.6340067788074, + "usage": { + "prompt_tokens": 65536, + "total_tokens": 65537, + "completion_tokens": 1, + "prompt_tokens_details": { + "cached_tokens": 0, + "created_cache_tokens": 64512 + }, + "completion_tokens_details": { + "reasoning_tokens": 1 + } + }, + "phase": "measured", + "sample": 0 + }, + { + "tokens": 8192, + "ttft_seconds": 2.812999999994645, + "tokens_per_second": 2912.1933878477053, + "usage": { + "prompt_tokens": 8192, + "total_tokens": 8193, + "completion_tokens": 1, + "prompt_tokens_details": { + "cached_tokens": 0, + "created_cache_tokens": 7168 + }, + "completion_tokens_details": { + "reasoning_tokens": 1 + } + }, + "phase": "measured", + "sample": 1 + }, + { + "tokens": 16384, + "ttft_seconds": 5.672000000005937, + "tokens_per_second": 2888.575458389078, + "usage": { + "prompt_tokens": 16384, + "total_tokens": 16385, + "completion_tokens": 1, + "prompt_tokens_details": { + "cached_tokens": 0, + "created_cache_tokens": 15360 + }, + "completion_tokens_details": { + "reasoning_tokens": 1 + } + }, + "phase": "measured", + "sample": 1 + }, + { + "tokens": 32768, + "ttft_seconds": 11.328000000008615, + "tokens_per_second": 2892.6553672294385, + "usage": { + "prompt_tokens": 32768, + "total_tokens": 32769, + "completion_tokens": 1, + "prompt_tokens_details": { + "cached_tokens": 0, + "created_cache_tokens": 31744 + }, + "completion_tokens_details": { + "reasoning_tokens": 1 + } + }, + "phase": "measured", + "sample": 1 + }, + { + "tokens": 65536, + "ttft_seconds": 22.718000000008033, + "tokens_per_second": 2884.7609824798324, + "usage": { + "prompt_tokens": 65536, + "total_tokens": 65537, + "completion_tokens": 1, + "prompt_tokens_details": { + "cached_tokens": 0, + "created_cache_tokens": 64512 + }, + "completion_tokens_details": { + "reasoning_tokens": 1 + } + }, + "phase": "measured", + "sample": 1 + }, + { + "tokens": 8192, + "ttft_seconds": 2.7969999999913853, + "tokens_per_second": 2928.8523418038008, + "usage": { + "prompt_tokens": 8192, + "total_tokens": 8193, + "completion_tokens": 1, + "prompt_tokens_details": { + "cached_tokens": 0, + "created_cache_tokens": 7168 + }, + "completion_tokens_details": { + "reasoning_tokens": 1 + } + }, + "phase": "measured", + "sample": 2 + }, + { + "tokens": 16384, + "ttft_seconds": 5.671999999991385, + "tokens_per_second": 2888.5754583964886, + "usage": { + "prompt_tokens": 16384, + "total_tokens": 16385, + "completion_tokens": 1, + "prompt_tokens_details": { + "cached_tokens": 0, + "created_cache_tokens": 15360 + }, + "completion_tokens_details": { + "reasoning_tokens": 1 + } + }, + "phase": "measured", + "sample": 2 + }, + { + "tokens": 32768, + "ttft_seconds": 11.343999999997322, + "tokens_per_second": 2888.5754583927833, + "usage": { + "prompt_tokens": 32768, + "total_tokens": 32769, + "completion_tokens": 1, + "prompt_tokens_details": { + "cached_tokens": 0, + "created_cache_tokens": 31744 + }, + "completion_tokens_details": { + "reasoning_tokens": 1 + } + }, + "phase": "measured", + "sample": 2 + }, + { + "tokens": 65536, + "ttft_seconds": 22.75, + "tokens_per_second": 2880.703296703297, + "usage": { + "prompt_tokens": 65536, + "total_tokens": 65537, + "completion_tokens": 1, + "prompt_tokens_details": { + "cached_tokens": 0, + "created_cache_tokens": 64512 + }, + "completion_tokens_details": { + "reasoning_tokens": 1 + } + }, + "phase": "measured", + "sample": 2 + } + ] +} \ No newline at end of file diff --git a/performance/records/glm53-flash/continuation-checkpoints-20260906/continuation-recurrence.json b/performance/records/glm53-flash/continuation-checkpoints-20260906/continuation-recurrence.json new file mode 100644 index 00000000..64a43b68 --- /dev/null +++ b/performance/records/glm53-flash/continuation-checkpoints-20260906/continuation-recurrence.json @@ -0,0 +1,47 @@ +{ + "gpu_executed": true, + "kernel_source_pins": { + "b12x.sequence.kda_prefill._impl": "0d0fc64b5fddd269d823a57f14c09a9e0a1b38a4f802c39df07353f34873a270", + "b12x.sequence.kda_prefill._cute_kernels": "df12ee84e1677ce1a200d1aab488c9ce5ae774d21f70b7f15b39c636623ab753", + "b12x.sequence.kda_prefill._policy": "ec64f07f841675501ff279bae3074d374ca8015a982ab12dc837ea2d03720a8e", + "b12x.sequence.kda_prefill.metadata": "7a9ef9ef3c59c88e4816bea6c9a74ebadd91a85632b42e221a5dd4009c5bb65c" + }, + "cases": [ + { + "tail_tokens": 2048, + "inplace": false, + "pass": true + }, + { + "tail_tokens": 2048, + "inplace": true, + "pass": true + }, + { + "tail_tokens": 4096, + "inplace": false, + "pass": true + }, + { + "tail_tokens": 4096, + "inplace": true, + "pass": true + }, + { + "tail_tokens": 8192, + "inplace": false, + "pass": true + }, + { + "tail_tokens": 8192, + "inplace": true, + "pass": true + } + ], + "pass": true, + "limitations": [ + "no convolution check", + "no scheduler/refcount integration", + "no full-model correctness or speed result" + ] +} \ No newline at end of file diff --git a/performance/records/glm53-flash/continuation-checkpoints-20260906/provenance.json b/performance/records/glm53-flash/continuation-checkpoints-20260906/provenance.json new file mode 100644 index 00000000..a3ca7c13 --- /dev/null +++ b/performance/records/glm53-flash/continuation-checkpoints-20260906/provenance.json @@ -0,0 +1,28 @@ +[ + { + "public_file": "continuation-prefill.json", + "sha256": "78a35a51a0b851cba7ad56307e54a2316b8ce2e3ce69a2339919b7d575f5e128", + "transformation": "None: exact source bytes; private source location omitted." + }, + { + "public_file": "continuation-prefill-128k.json", + "sha256": "3ba7cb58ea757905e4e164e390f6409418bf04d5451146190b68fb2dcce9a251", + "transformation": "None: exact source bytes; private source location omitted." + }, + { + "public_file": "continuation-recurrence.json", + "sha256": "5ff46ae9c343e889d7319b7f0db3fafe28bea79762c1571ca64136974ed3cdbf", + "transformation": "None: exact source bytes; private source location omitted." + }, + { + "public_file": "continuation-convolution.json", + "sha256": "42ff4bbf70ce2d649a95af219447b3c5a262539bca8dece06205a7c3ab2dbc36", + "transformation": "None: exact source bytes; private source location omitted." + }, + { + "public_file": "../../../harnesses/vllm/prefill_checks/continuation_serve_checks.py", + "historical_harness_sha256": "f8c8d0f75b13ed833371bbef89c4e47a89ec545dfa39135df4969f39ec0f4f05", + "public_harness_sha256": "73ec5e4ad756da7bd6d105f5658bc038d3989f2dbdb699873740426f5572c711", + "transformation": "Endpoint/model/SSH alias and container selection parameterized; os import added where required, unused import removed, UTF-8 output made explicit, source formatted. Timing/request logic retained. Public copy has not been rerun on GPU. Historical source digest was computed from a retained copy, not signed at execution time." + } +] diff --git a/runtime/glm53-spark-mtp3-mesh/performance/continuation/README.md b/runtime/glm53-spark-mtp3-mesh/performance/continuation/README.md index 51fa27b1..f5c48c05 100644 --- a/runtime/glm53-spark-mtp3-mesh/performance/continuation/README.md +++ b/runtime/glm53-spark-mtp3-mesh/performance/continuation/README.md @@ -4,6 +4,10 @@ Status: **research-only**. The source package preserves the four continuation files used by the serving image identified below. CPU tests verify package integrity and ownership updates; they do not qualify the combined rebuilt image. +[Bounded serving evidence](../../../../performance/records/glm53-flash/continuation-checkpoints-20260906/README.md) +records cold 8K–128K timing, GPU state checks, cache-reuse limits, and the exact +image configuration. Those observations have no paired-control speedup claim. + `source.tar.gz` contains exact vLLM source bytes for: - `vllm/v1/core/kv_cache_manager.py` @@ -62,3 +66,7 @@ python -m pytest runtime/glm53-spark-mtp3-mesh/performance/continuation -q The tests reject altered packages, unexpected runtime preimages, and ownership drift before any source write. Attribution tests execute scheduler boundaries against both the fresh-prompt and continuation source variants. +The runtime regressions execute packaged scheduler/allocator methods to cover +source retention, speculative-reserve overlap, allocation failure, publication +pins, worker-visible block IDs, admission provenance, and cleanup. The 6K +scheduling case is CPU-only coverage, not serving qualification at that size. diff --git a/runtime/glm53-spark-mtp3-mesh/performance/continuation/test_runtime.py b/runtime/glm53-spark-mtp3-mesh/performance/continuation/test_runtime.py new file mode 100644 index 00000000..109d4fe6 --- /dev/null +++ b/runtime/glm53-spark-mtp3-mesh/performance/continuation/test_runtime.py @@ -0,0 +1,619 @@ +"""CPU tests of candidate methods, not live model or CUDA qualification.""" + +import ast +from collections import defaultdict +from dataclasses import dataclass +import importlib.util +from pathlib import Path +from types import SimpleNamespace as NS, ModuleType +import sys +import unittest +from unittest.mock import patch + +ROOT = Path(__file__).parent +spec = importlib.util.spec_from_file_location( + "continuation_regression_installer", ROOT / "install.py" +) +installer = importlib.util.module_from_spec(spec) +spec.loader.exec_module(installer) +_, sources = installer.package() +helper = ModuleType("continuation_regression_helper") +exec( + compile( + sources["vllm/v1/core/recurrent_prefill_checkpoint.py"], + "recurrent_prefill_checkpoint.py", + "exec", + ), + helper.__dict__, +) + + +class MambaSpec: + block_size = 512 + num_prefill_checkpoint_blocks = 2 + + +def extract(path, name, methods, scope, base=None): + cls = next( + n + for n in ast.parse(sources["vllm/" + path]).body + if isinstance(n, ast.ClassDef) and n.name == name + ) + cls.body = [ + n + for n in cls.body + if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef)) and n.name in methods + ] + cls.bases = [ast.Name(id=base, ctx=ast.Load())] if base else [] + cls.decorator_list = [] + module = ast.Module( + body=[ + ast.ImportFrom( + module="__future__", names=[ast.alias(name="annotations")], level=0 + ), + cls, + ], + type_ignores=[], + ) + exec(compile(ast.fix_missing_locations(module), str(ROOT / path), "exec"), scope) + return scope[name] + + +@dataclass +class Block: + block_id: int + is_null: bool = False + ref_cnt: int = 1 + block_hash: object = None + block_hash_num_tokens: int = 0 + state: object = None + + +class Pool: + hash_block_size = 512 + + def __init__(self): + self.next = 1 + self.allocated = [] + self.freed = [] + self.fail = False + self.registered = [] + + def get_new_blocks(self, count): + if self.fail: + raise RuntimeError("injected pool failure") + blocks = [Block(i) for i in range(self.next, self.next + count)] + self.next += count + self.allocated.extend(blocks) + return blocks + + def free_blocks(self, blocks): + for block in blocks: + if block.is_null: + continue + assert block.ref_cnt > 0 + block.ref_cnt -= 1 + self.freed.append(block.block_id) + + def cache_full_blocks( + self, + *, + request, + blocks, + num_cached_blocks, + num_full_blocks, + block_size, + kv_cache_group_id, + block_mask, + ): + for column in range(num_cached_blocks, num_full_blocks): + block = blocks[column] + if block.is_null or ( + block_mask is not None and not block_mask[column - num_cached_blocks] + ): + continue + block.block_hash = (column, kv_cache_group_id) + block.block_hash_num_tokens = (column + 1) * block_size + self.registered.append( + (column, block.block_id, block.block_hash_num_tokens) + ) + + +def manager(speculative=3): + path = "v1/core/single_type_kv_cache_manager.py" + scope = { + "MambaSpec": MambaSpec, + "cdiv": lambda a, b: (a + b - 1) // b, + "continuation_layout": helper.continuation_layout, + "get_group_id": lambda key: key[1], + } + base = extract( + path, + "SingleTypeKVCacheManager", + { + "cache_blocks", + "remove_skipped_blocks", + "_remove_blocks_in_range", + "pop_blocks_for_free", + "free", + }, + scope, + ) + scope["Base"] = base + cls = extract( + path, + "MambaManager", + { + "get_num_blocks_to_allocate", + "allocate_new_blocks", + "remove_skipped_blocks", + "_needs_internal_checkpoint", + "get_num_skipped_tokens", + "cache_blocks", + "_cache_partial_tail_block", + "_queue_aligned_recurrent_boundary", + "reachable_block_mask", + "pop_blocks_for_free", + }, + scope, + base="Base", + ) + obj = cls() + obj.kv_cache_spec = MambaSpec() + obj.block_size = 512 + obj.num_speculative_blocks = speculative + obj.mamba_cache_mode = "align" + obj.req_to_blocks = defaultdict(list) + obj._planned_recurrent_checkpoints = {} + obj._planned_recurrent_publications = {} + obj._allocated_block_reqs = set() + obj._partial_hit_reqs = {} + obj._num_checkpoint_blocks = {} + obj.last_state_block_idx = {} + obj._null_block = Block(0, True, 0) + obj.block_pool = Pool() + obj.num_cached_block = {} + obj.cached_blocks_this_step = set() + obj.scheduler_block_size = 2048 + obj.use_eagle = True + obj.kv_cache_group_id = 7 + obj._pending_aligned_recurrent_boundaries = [] + obj.recurrent_publication_boundary = None + obj._producer_partial_tail_reqs = {} + obj._pending_partial_tail_offloads = [] + obj._has_partial_local_hit = lambda *_: False + obj._get_num_evictable_blocks = lambda _: 0 + return obj + + +def seed(obj, start): + worker = [] + for end in range(8192, start + 1, 8192): + obj.remove_skipped_blocks("r", max(0, end - 8192)) + suffix = obj.allocate_new_blocks("r", end, end) + worker.extend(suffix) + obj.req_to_blocks["r"][end // 512 - 1].state = end + return worker + + +def request(prompt, start=0, identifier="r"): + return NS( + request_id=identifier, + num_computed_tokens=start, + num_prompt_tokens=prompt, + num_tokens=prompt, + shared_prefix_boundary=0, + has_encoder_inputs=False, + num_preemptions=0, + spec_token_ids=[], + num_in_flight_tokens=0, + status="waiting", + resumable=False, + ) + + +def scheduler(obj, enabled=True, budget=8192): + scope = { + "fresh_prompt_plan": helper.fresh_prompt_plan, + "final_chunk_plan": helper.final_chunk_plan, + "continuation_layout": helper.continuation_layout, + "MambaSpec": MambaSpec, + "RequestStatus": NS(WAITING="waiting"), + } + cls = extract( + "v1/core/sched/scheduler.py", + "Scheduler", + { + "_record_continuation_origin", + "_recurrent_checkpoint_plan", + "_mamba_block_aligned_split", + "_free_request_blocks", + }, + scope, + ) + owner = cls() + owner._two_checkpoint_prefill_enabled = True + owner._continuation_prefill_enabled = enabled + owner._continuation_prefill_origins = {} + owner.cache_config = NS(block_size=512) + owner.max_num_scheduled_tokens = budget + owner.use_eagle = True + owner.mamba_has_prefill_checkpoint_blocks = True + owner.mamba_partial_cache_hit = True + owner.hash_block_size = 512 + owner.scheduler_config = NS(long_prefill_token_threshold=0) + owner.kv_cache_manager = NS(coordinator=NS(single_type_managers=[obj])) + owner._recurrent_publication_boundaries = lambda req: tuple( + [((req.num_prompt_tokens - 1) // 2048) * 2048] + ) + return owner + + +class ContinuationTests(unittest.TestCase): + def test_fresh_8k_path_retained_when_continuation_disabled(self): + obj = manager() + owner = scheduler(obj, enabled=False) + req = request(8192) + plan = owner._recurrent_checkpoint_plan(req, 0, 8192) + self.assertEqual(plan, (0, 8192, (6144, 7168))) + obj._planned_recurrent_checkpoints["r"] = plan + self.assertEqual(obj.get_num_blocks_to_allocate("r", 8192, [], 0, 0, 8192), 6) + table = obj.allocate_new_blocks("r", 8192, 8192) + self.assertEqual( + [i for i, b in enumerate(table) if not b.is_null], [11, 13, 15, 16, 17, 18] + ) + + def test_request_release_keeps_an_extra_publication_pin(self): + for end, targets in ((10240, (9216,)), (16384, (14336, 15360))): + obj = manager() + seed(obj, 8192) + obj._planned_recurrent_checkpoints["r"] = (8192, end, targets) + obj.allocate_new_blocks("r", end, end) + published = obj.req_to_blocks["r"][targets[0] // 512 - 1] + published.state = targets[0] + published.ref_cnt += 1 + obj.free("r") + self.assertEqual(published.ref_cnt, 1) + self.assertEqual(published.state, targets[0]) + self.assertNotIn("r", obj.req_to_blocks) + self.assertNotIn("r", obj.last_state_block_idx) + self.assertEqual(len(obj.block_pool.freed), len(set(obj.block_pool.freed))) + obj.block_pool.free_blocks([published]) + self.assertEqual(published.ref_cnt, 0) + + def test_real_scheduler_and_allocator_schedules_8k_and_6k(self): + expected = { + 8192: { + 9216: [8192, 1024], + 10240: [8192, 2048], + 16384: [8192, 8192], + 32768: [8192] * 4, + }, + 6144: { + 9216: [6144, 3072], + 10240: [6144, 4096], + 16384: [6144, 6144, 4096], + 32768: [6144] * 5 + [2048], + }, + } + for budget, cases in expected.items(): + for prompt, want in cases.items(): + with self.subTest(budget=budget, prompt=prompt): + obj = manager() + owner = scheduler(obj, budget=budget) + req = request(prompt) + chunks = [] + worker = [] + while req.num_computed_tokens < prompt: + start = req.num_computed_tokens + size = owner._mamba_block_aligned_split( + req, min(budget, prompt - start) + ) + plan = owner._recurrent_checkpoint_plan( + req, start, start + size + ) + obj.remove_skipped_blocks("r", start) + if plan: + obj._planned_recurrent_checkpoints["r"] = plan + count = obj.get_num_blocks_to_allocate( + "r", start + size, [], start, start, start + size + ) + before = len(obj.block_pool.allocated) + worker.extend( + obj.allocate_new_blocks("r", start + size, start + size) + ) + self.assertEqual(len(obj.block_pool.allocated) - before, count) + # Simulated completion stamps only states this step actually produces. + if plan: + for token in plan[2]: + obj.req_to_blocks["r"][token // 512 - 1].state = token + obj.req_to_blocks["r"][(start + size) // 512 - 1].state = ( + start + size + ) + if start == 0: + owner._record_continuation_origin(req, 0, 0, 0, False) + obj._planned_recurrent_checkpoints.clear() + chunks.append(size) + req.num_computed_tokens += size + self.assertEqual(chunks, want) + self.assertEqual(worker[prompt // 512 - 1].state, prompt) + + def test_actual_allocator_source_and_worker_prefix(self): + for speculative in (0, 1, 3): + for start in (8192, 16384, 24576, 57344): + for tail in (1536, 2048, 4096, 6144, 8192): + with self.subTest(speculative=speculative, start=start, tail=tail): + obj = manager(speculative) + worker = seed(obj, start) + old = tuple(obj.req_to_blocks["r"]) + worker_before = tuple(worker) + plan = helper.final_chunk_plan( + start=start, + end=start + tail, + prompt=start + tail, + num_tokens=start + tail, + block_size=512, + publications=(((start + tail - 1) // 2048) * 2048,), + ) + obj._planned_recurrent_checkpoints["r"] = plan + admission = obj.get_num_blocks_to_allocate( + "r", start + tail, [], start, start, start + tail + ) + self.assertEqual(admission, len(plan[2]) + 1) + before = len(obj.block_pool.allocated) + suffix = obj.allocate_new_blocks( + "r", start + tail, start + tail + ) + worker.extend(suffix) + self.assertEqual( + len(obj.block_pool.allocated) - before, admission + ) + self.assertEqual( + worker[: len(worker_before)], list(worker_before) + ) + table = obj.req_to_blocks["r"] + source = start // 512 - 1 + self.assertIs(table[source], old[source]) + self.assertEqual(table[source].state, start) + active = ( + [source] + + [p // 512 - 1 for p in plan[2]] + + list(range((start + tail) // 512 - 1, len(table))) + ) + self.assertEqual( + len(active), len({worker[c].block_id for c in active}) + ) + self.assertTrue(all(worker[c] is table[c] for c in active)) + owned = [b.block_id for b in table if not b.is_null] + self.assertEqual(len(owned), len(set(owned))) + self.assertEqual(obj.last_state_block_idx["r"], source) + # An in-flight next chunk must not free its source yet. + obj.remove_skipped_blocks("r", start) + self.assertIs(obj.req_to_blocks["r"][source], old[source]) + # After that chunk is processed, ordinary cleanup may retire it. + obj.remove_skipped_blocks("r", start + tail) + self.assertTrue(obj.req_to_blocks["r"][source].is_null) + + def test_10k_reuses_existing_checkpoint_column(self): + obj = manager() + worker = seed(obj, 8192) + checkpoint = worker[17] + plan = (8192, 10240, (9216,)) + obj._planned_recurrent_checkpoints["r"] = plan + worker.extend(obj.allocate_new_blocks("r", 10240, 10240)) + self.assertIs(obj.req_to_blocks["r"][17], checkpoint) + self.assertIs(worker[17], checkpoint) + self.assertEqual(len(obj.block_pool.allocated), 6) # initial4 +new2 + + def test_failed_pool_allocation_does_not_mutate_table(self): + obj = manager() + seed(obj, 8192) + before = tuple(obj.req_to_blocks["r"]) + obj.block_pool.fail = True + obj._planned_recurrent_checkpoints["r"] = (8192, 16384, (14336, 15360)) + with self.assertRaises(RuntimeError): + obj.allocate_new_blocks("r", 16384, 16384) + self.assertEqual(tuple(obj.req_to_blocks["r"]), before) + self.assertNotIn("r", obj.last_state_block_idx) + + def test_private_reserve_and_source_guards(self): + for mutate in ( + lambda o: setattr(o.req_to_blocks["r"][16], "ref_cnt", 2), + lambda o: setattr(o.req_to_blocks["r"][17], "block_hash", ("x", 7)), + lambda o: o.req_to_blocks["r"].__setitem__(15, o._null_block), + lambda o: o.req_to_blocks["r"].__setitem__(18, o.req_to_blocks["r"][17]), + lambda o: o.req_to_blocks["r"].append(o._null_block), + ): + obj = manager() + seed(obj, 8192) + mutate(obj) + before = tuple(obj.req_to_blocks["r"]) + with self.assertRaises(ValueError): + helper.continuation_layout( + before, (8192, 16384, (14336, 15360)), 512, 3 + ) + self.assertEqual(tuple(obj.req_to_blocks["r"]), before) + + def test_scheduler_opt_in_provenance_and_fallback(self): + obj = manager() + owner = scheduler(obj) + req = request(16384) + self.assertIsNone(owner._recurrent_checkpoint_plan(req, 8192, 16384)) + owner._record_continuation_origin(req, 0, 0, 0, False) + seed(obj, 8192) + req.num_computed_tokens = 8192 + self.assertEqual(owner._mamba_block_aligned_split(req, 8192), 8192) + owner._continuation_prefill_enabled = False + self.assertEqual(owner._mamba_block_aligned_split(req, 8192), 6144) + owner._continuation_prefill_enabled = True + for field, value in [ + ("num_preemptions", 1), + ("has_encoder_inputs", True), + ("spec_token_ids", [3]), + ("num_tokens", 16385), + ("resumable", True), + ]: + old = getattr(req, field) + setattr(req, field, value) + self.assertIsNone(owner._recurrent_checkpoint_plan(req, 8192, 16384)) + setattr(req, field, old) + replacement = request(16384, 8192) + self.assertIsNone(owner._recurrent_checkpoint_plan(replacement, 8192, 16384)) + + def test_origin_rejected_for_cached_or_async_admission_and_cleared_on_free(self): + obj = manager() + owner = scheduler(obj) + req = request(16384) + for args in [(6144, 0, 6144, False), (6144, 6144, 0, False), (0, 0, 0, True)]: + owner._record_continuation_origin(req, *args) + self.assertNotIn("r", owner._continuation_prefill_origins) + owner._record_continuation_origin(req, 0, 0, 0, False) + owner.defer_block_free = False + owner.kv_cache_manager.free = lambda _: None + owner._free_request_blocks(req) + self.assertNotIn("r", owner._continuation_prefill_origins) + + def test_6k_budget_and_hard_cap(self): + self.assertIsNone( + helper.final_chunk_plan( + start=8192, + end=16384, + prompt=16384, + num_tokens=16384, + block_size=512, + publications=(14336,), + max_chunk_tokens=6144, + ) + ) + self.assertEqual( + helper.final_chunk_plan( + start=12288, + end=16384, + prompt=16384, + num_tokens=16384, + block_size=512, + publications=(14336,), + max_chunk_tokens=6144, + ), + (12288, 16384, (14336, 15360)), + ) + self.assertIsNone( + helper.final_chunk_plan( + start=8192, + end=24576, + prompt=24576, + num_tokens=24576, + block_size=512, + publications=(22528,), + max_chunk_tokens=32768, + ) + ) + + def test_cache_registration_and_interior_publication(self): + obj = manager() + seed(obj, 8192) + obj.num_cached_block["r"] = 16 + plan = (8192, 16384, (14336, 15360)) + obj._planned_recurrent_checkpoints["r"] = plan + obj._planned_recurrent_publications["r"] = (14336,) + obj.allocate_new_blocks("r", 16384, 16384) + req = request(16384, 8192) + obj.cache_blocks(req, 16384) + by_position = { + position: identifier + for _, identifier, position in obj.block_pool.registered + } + self.assertEqual(by_position[14336], obj.req_to_blocks["r"][27].block_id) + self.assertEqual(by_position[15360], obj.req_to_blocks["r"][29].block_id) + self.assertEqual( + [row[3] for row in obj._pending_aligned_recurrent_boundaries], [14336] + ) + + def test_wrapper_clears_plan_on_failed_admission(self): + obj = manager() + seed(obj, 8192) + req = request(16384, 8192) + cls = extract( + "v1/core/kv_cache_manager.py", "KVCacheManager", {"allocate_slots"}, {} + ) + owner = cls() + owner.coordinator = NS(single_type_managers=[obj]) + observed = [] + + def inner(**kwargs): + observed.append(obj._planned_recurrent_checkpoints["r"]) + return None + + owner._allocate_slots_without_checkpoint_plan = inner + with patch.dict( + sys.modules, {"vllm.v1.core.recurrent_prefill_checkpoint": helper} + ): + self.assertIsNone( + owner.allocate_slots( + req, + 8192, + recurrent_prefill_checkpoint_plan=(8192, 16384, (14336, 15360)), + recurrent_checkpoint_publications=(14336,), + ) + ) + self.assertEqual(len(observed), 1) + self.assertFalse(obj._planned_recurrent_checkpoints) + self.assertFalse(obj._planned_recurrent_publications) + + def raises(**kwargs): + raise RuntimeError("injected inner failure") + + owner._allocate_slots_without_checkpoint_plan = raises + with patch.dict( + sys.modules, {"vllm.v1.core.recurrent_prefill_checkpoint": helper} + ): + with self.assertRaises(RuntimeError): + owner.allocate_slots( + req, + 8192, + recurrent_prefill_checkpoint_plan=(8192, 16384, (14336, 15360)), + recurrent_checkpoint_publications=(14336,), + ) + for extra in ( + {"num_external_computed_tokens": 512}, + {"delay_cache_blocks": True}, + {"recurrent_checkpoint_publications": (8192,)}, + ): + with self.assertRaises(ValueError): + owner.allocate_slots( + req, + 8192, + recurrent_prefill_checkpoint_plan=(8192, 16384, (14336, 15360)), + **extra, + ) + self.assertFalse(obj._planned_recurrent_checkpoints) + self.assertFalse(obj._planned_recurrent_publications) + + def test_hooks_are_in_actual_admission_and_cleanup_sites(self): + tree = ast.parse(sources["vllm/v1/core/sched/scheduler.py"]) + methods = {n.name: n for n in ast.walk(tree) if isinstance(n, ast.FunctionDef)} + calls = [ + n + for n in ast.walk(methods["schedule"]) + if isinstance(n, ast.Call) + and isinstance(n.func, ast.Attribute) + and n.func.attr == "_record_continuation_origin" + ] + self.assertEqual(len(calls), 1) + self.assertIn( + "_continuation_prefill_origins.pop", + ast.unparse(methods["_update_requests_with_invalid_blocks"]), + ) + for call in [ + n for n in ast.walk(methods["schedule"]) if isinstance(n, ast.Call) + ]: + for keyword in call.keywords: + if keyword.arg == "recurrent_checkpoint_publications": + self.assertIn( + "checkpoint_plan[0] < p < checkpoint_plan[1]", + ast.unparse(keyword.value), + ) + + +if __name__ == "__main__": + unittest.main(verbosity=2) From 0e3931e0ceedb1e29322bedbd44e71521b5c98a6 Mon Sep 17 00:00:00 2001 From: FujitsuPolycom <87842395+FujitsuPolycom@users.noreply.github.com> Date: Sun, 6 Sep 2026 22:41:31 -0500 Subject: [PATCH 5/5] Require explicit cache accounting in continuation measurements Reject absent, malformed, boolean or negative cached-token counts instead of accepting a missing field as zero. Apply the same accounting boundary to semantic reuse checks and cold timing. Preserve all raw observations: all 19 retained continuation rows contain explicit integer zero counts. Update portable-harness provenance. Validation: 14 offline readiness/timing tests pass, including six reproductions of missing or invalid accounting; Ruff passed. Runtime source packages are unchanged. --- .../continuation_serve_checks.py | 13 +++++- .../prefill_checks/test_prefill_checks.py | 40 +++++++++++++++++++ .../README.md | 5 ++- .../provenance.json | 4 +- 4 files changed, 57 insertions(+), 5 deletions(-) diff --git a/performance/harnesses/vllm/prefill_checks/continuation_serve_checks.py b/performance/harnesses/vllm/prefill_checks/continuation_serve_checks.py index 39921df9..b2245e2f 100644 --- a/performance/harnesses/vllm/prefill_checks/continuation_serve_checks.py +++ b/performance/harnesses/vllm/prefill_checks/continuation_serve_checks.py @@ -17,6 +17,15 @@ raise ValueError("Unexpected container name") +def cached_prompt_tokens(usage): + """Missing accounting cannot establish cold work or successful cache reuse.""" + details = usage.get("prompt_tokens_details") + cached = details.get("cached_tokens") if isinstance(details, dict) else None + if type(cached) is not int or cached < 0: + raise ValueError("Usage must provide a nonnegative integer cached_tokens") + return cached + + def get(path): with urllib.request.urlopen(BASE + path, timeout=10) as r: return r.read().decode() @@ -114,7 +123,7 @@ def semantic(tokens, fact): choice = response["choices"][0] answer = (choice["message"].get("content") or "").strip() usage = response["usage"] - cached = usage.get("prompt_tokens_details", {}).get("cached_tokens", 0) + cached = cached_prompt_tokens(usage) row = { "tokens": tokens, "phase": phase, @@ -176,7 +185,7 @@ def prefill(tokens): if first is None or usage is None: raise RuntimeError("No streamed token or final usage") assert usage["prompt_tokens"] == tokens - assert usage.get("prompt_tokens_details", {}).get("cached_tokens", 0) == 0 + assert cached_prompt_tokens(usage) == 0 return { "tokens": tokens, "ttft_seconds": first, diff --git a/performance/harnesses/vllm/prefill_checks/test_prefill_checks.py b/performance/harnesses/vllm/prefill_checks/test_prefill_checks.py index d8f8e1b0..1cf661c2 100644 --- a/performance/harnesses/vllm/prefill_checks/test_prefill_checks.py +++ b/performance/harnesses/vllm/prefill_checks/test_prefill_checks.py @@ -95,3 +95,43 @@ def test_timing_rejects_wrong_prompt_length_or_cached_work( result = harness.prefill(8192) assert result["ttft_seconds"] == 0.25 assert result["tokens_per_second"] == 32768 + + +@pytest.mark.parametrize( + "details", + [ + "omitted", + None, + {}, + {"cached_tokens": False}, + {"cached_tokens": "0"}, + {"cached_tokens": -1}, + ], +) +def test_timing_rejects_unproven_cache_accounting(harness, monkeypatch, details): + clock = iter([10.0, 10.25]) + clock_name = ( + "perf_counter" + if harness.__name__.endswith("mhc_precise_checks") + else "monotonic" + ) + monkeypatch.setattr(harness.time, clock_name, lambda: next(clock)) + monkeypatch.setattr( + harness, "calibrate", lambda *args: [{"role": "user", "content": "test"}] + ) + usage = {"prompt_tokens": 8192} + if details != "omitted": + usage["prompt_tokens_details"] = details + chunks = [ + {"choices": [{"delta": {"content": "answer"}}]}, + {"choices": [], "usage": usage}, + ] + stream = ( + b"".join(b"data: " + json.dumps(chunk).encode() + b"\n" for chunk in chunks) + + b"data: [DONE]\n" + ) + monkeypatch.setattr( + harness.urllib.request, "urlopen", lambda *a, **kw: io.BytesIO(stream) + ) + with pytest.raises(ValueError, match="cached_tokens"): + harness.prefill(8192) diff --git a/performance/records/glm53-flash/continuation-checkpoints-20260906/README.md b/performance/records/glm53-flash/continuation-checkpoints-20260906/README.md index e1e85407..80921a06 100644 --- a/performance/records/glm53-flash/continuation-checkpoints-20260906/README.md +++ b/performance/records/glm53-flash/continuation-checkpoints-20260906/README.md @@ -81,7 +81,10 @@ startup, and serving validation. ## Reproduction The [portable harness](../../../harnesses/vllm/prefill_checks/continuation_serve_checks.py) -retains the historical timing and request logic. Set `BENCH_API_BASE`, +retains the historical timing and request logic and additionally rejects +missing or malformed cache-token accounting. Every retained timing row contains +explicit integer cache accounting, so this check does not change the record. +Set `BENCH_API_BASE`, `BENCH_MODEL`, `BENCH_RANK0_SSH`, and `BENCH_RANK0_CONTAINER` for an explicitly authorized idle deployment, then run: diff --git a/performance/records/glm53-flash/continuation-checkpoints-20260906/provenance.json b/performance/records/glm53-flash/continuation-checkpoints-20260906/provenance.json index a3ca7c13..1e677bf1 100644 --- a/performance/records/glm53-flash/continuation-checkpoints-20260906/provenance.json +++ b/performance/records/glm53-flash/continuation-checkpoints-20260906/provenance.json @@ -22,7 +22,7 @@ { "public_file": "../../../harnesses/vllm/prefill_checks/continuation_serve_checks.py", "historical_harness_sha256": "f8c8d0f75b13ed833371bbef89c4e47a89ec545dfa39135df4969f39ec0f4f05", - "public_harness_sha256": "73ec5e4ad756da7bd6d105f5658bc038d3989f2dbdb699873740426f5572c711", - "transformation": "Endpoint/model/SSH alias and container selection parameterized; os import added where required, unused import removed, UTF-8 output made explicit, source formatted. Timing/request logic retained. Public copy has not been rerun on GPU. Historical source digest was computed from a retained copy, not signed at execution time." + "public_harness_sha256": "8b7b1cb03a1967317e3d02f21a1a6dbb5b89e29a434a1dff5f7069d50175e962", + "transformation": "Endpoint/model/SSH alias and container selection parameterized; os import added where required, unused import removed, UTF-8 output made explicit, source formatted. Timing/request logic retained; missing or malformed cache-token accounting explicitly rejected. All retained timing rows contain explicit integer cached_tokens=0. Public copy has not been rerun on GPU. Historical source digest was computed from a retained copy, not signed at execution time." } ]