[codex] Optimize dynamic organ lifecycle refresh

[VEZY]

Summary

This PR reduces the lifecycle-refresh overhead introduced by dynamic organ creation on the multi-plants runtime path. It keeps the compiled model plan and public API unchanged while updating only the object-dependent runtime structures that actually change.

Problem

The 4,160-day XPalm reference simulation currently takes about 14.5 seconds on the exact multi-plants base, even though all models use the same cadence and the model application plan is fixed. Profiling showed that a large share of the runtime was spent refreshing runtime structures after organs were added.

The main avoidable cost was eager construction of complete nested Many hard-call execution targets at every lifecycle barrier. XPalm only executes the call for the newly created object in this path, but the runtime still materialized 61,592 nested targets that were never used. Other object-dependent carriers, previous-timestep views, distributed-output destinations, and call-owner batches were also being rebuilt more broadly than needed.

Changes

• Add lazy execution batches for Many hard-call targets while keeping One and OptionalOne targets eager and concretely typed.
• Incrementally extend Many input carriers, previous-timestep views, distributed-output destinations and streams, and hard-call owner/batch structures when objects are only added.
• Preserve the safe full-rebuild path for removals, reparenting, and other complex lifecycle changes.
• Install final direct input references without first constructing default Status values that are immediately replaced. This avoids 686 redundant rewrites in the XPalm reference while preserving reference identity, conversions, callbacks, diagnostics, and error behavior.
• Keep environment-binding invalidation correct for lifecycle changes.

There is no public API change.

User impact and benchmarks

Warm, like-for-like runs on the same machine:

• Exact multi-plants base: about 14.50 s wall time / 18.94 s process CPU.
• This branch: about 7.4-7.8 s wall time / 8.7-8.8 s process CPU.

The optimized XPalm run remains bit-for-bit identical to the exact base for the retained benchmark outputs:

• SHA-256 digest: 1e47d0a88fc5742ad24b6a4fbbaa6006b047fcc56eaa4f54f468bb8b60c66b6c
• Final step: 4160
• Phytomer count: 344
• LAI: 5.2085061095890355
• FTSW: 0.7991179101191208

The older PlantSimEngine v0.14.1 / XPalm v0.6.1 reference was around 4.41 s, but its scientific outputs differ, so that number is included only as historical context rather than a direct performance comparison.

PlantBiophysics was also checked over 8,760 steps and 113,880 output rows. The base and this branch produce the same digest (a8aad9a50aab94588d68d4e2b9b919821af7b239ed2d94c78c069c5b04c0a377), with no material timing or allocation regression.

Validation

• Unified model/object API: 747 / 747
• API stabilization: 644 / 644
• Status-type lifecycle: 49 / 49
• Hard calls: 139 / 139
• Previous-timestep views: 75 / 75
• Bound Many behavior: 68 / 68
• Distributed outputs: 84 / 84
• Initializers: 131 / 131
• Two independent static audits found no actionable issue.
• git diff --check passes.

The complete test/runtests.jl suite was not run: the Kaimon test project currently cannot start because MonteCarloMeasurements is declared as a direct test dependency but is absent from the test manifest, while the root-project runner does not provide Aqua. The focused suites above were run with their required imports in fresh Julia sessions.

[VEZY]

Current status: the first runtime-performance tranche is complete.

Profiling confirmed that XPalm follows the short, single-cadence path: the model application plan stays fixed, and only organs and their runtime state are added. The largest avoidable cost was therefore not model scheduling, but repeated lifecycle refresh work. In particular, the runtime eagerly materialized complete nested Many hard-call targets even though XPalm executes only the newly created object in this path; the reference run built 61,592 targets that were never used.

This branch now constructs Many call batches lazily and incrementally extends the object-dependent input carriers, previous-timestep views, distributed-output structures, and call-owner batches. It keeps the full-rebuild fallback for removals, reparenting, and complex changes. It also installs final direct input references without first creating default Status values that would immediately be replaced. There is no public API change.

On warm, like-for-like runs, the exact multi-plants base is about 14.50 s wall time / 18.94 s process CPU, while this branch is about 7.4-7.8 s wall time / 8.7-8.8 s process CPU. The retained XPalm outputs remain bit-for-bit identical (1e47d0a88fc5742ad24b6a4fbbaa6006b047fcc56eaa4f54f468bb8b60c66b6c). PlantBiophysics also keeps the exact same 8,760-step output digest, with no material timing or allocation regression.

The focused validation is green, including the 747-test unified API suite, the 644-test API stabilization suite, lifecycle, hard-call, previous-timestep, bound-Many, distributed-output, and initializer tests. The complete test/runtests.jl suite was not run because the current test environment cannot start with its manifest (MonteCarloMeasurements is missing), while the root runner does not provide Aqua; this limitation is also recorded in the PR description.

My proposed next step is to compile an internal, canonical object/status construction recipe from the fixed application plan. A newly emitted organ could then build its Status once from that recipe instead of passing through several output-default-binding reconstruction phases. The implementation must preserve Ref identity, conversion callbacks, errors, and diagnostics. This is currently expected to remain an internal optimization; I will stop and coordinate before proceeding if it requires any public API change.

[VEZY]

Dynamic-organ performance update

I completed another optimization pass and created local checkpoint f521b8db (perf: accelerate dynamic organ lifecycle). This checkpoint has not been pushed yet.

Results

• XPalm full 4,160-day run: 5.61–5.92 s, down from 7.15 s on the branch baseline (17–22% faster).
• Allocations: 2.95 GB, down from 4.35 GB (about 32% less).
• PlantBiophysics remained stable, with no observed regression in the no-retention, retained-output, construction, or fan-out profiles.
• No public API change.

What changed

• Reused canonical status recipes for dynamically emitted organs, while preserving arbitrary Ref aliases.
• Shared same-rate Many input storage and reduced selector work through bounded scope resolution.
• Replaced target-vector hashing with O(1) lifecycle membership generations.
• Added a transactional execution-plan suffix extension for pure object additions.
• Kept large manual Many hard-call memberships cold until they are actually observed, while retaining safe rebuilds after additions, removals, reparenting, and environment refreshes.
• Prevented diagnostics and graph rendering from materializing otherwise cold hard-call memberships.
• Preserved the zero-allocation hot path for materialized Many hard calls.

Correctness and validation

• Full retained-output comparison: 2,244 streams, exact digest 8eac9fef17914337d1343c3508739ac919a4f8d0cadb33fe1736e9b731b58d2d.
• Final XPalm state remained exactly:
  • current_step = 4160
  • phytomer_count = 344
  • lai = 5.2085061095890355
  • ftsw = 0.7991179101191208
• Focused validation passed:
  • organ lifecycle benchmark smoke: 33/33
  • hard-call benchmark smoke: 38/38
  • internal benchmark assembly: 32/32
  • complete model-graph focused suite

The next experiment is intentionally more invasive: construct a newborn organ's complete canonical status in one materialization through an internal pending-status overlay, instead of applying separate output and input phases. I will keep the public API unchanged and benchmark it independently before deciding whether to retain it.

[VEZY]

Performance update since the previous comment

Six additional commits have been pushed since checkpoint f521b8db:

• 492c521f appends stable Many input sources in place instead of rebuilding their storage.
• ce06cf26 caches lifecycle targets and runtime contexts across stable execution plans.
• d9ac84c2 reuses planned singleton input producers.
• 503f7a7d specializes the normal, uninstrumented execution path.
• 2dcedda4 indexes observed manual-call owners so lifecycle propagation does not scan unrelated bindings.
• b053534e refines that index to exact executable (callee application, object) targets, while preserving safe rebuilds for removals and reparenting and keeping unobserved large Many calls cold.

The overall design remains the same: the application plan is fixed, while only object-dependent targets and status are extended at lifecycle barriers. There is still no public API change.

Current performance and correctness

• XPalm full 4,160-day run, five warmed samples: 4.07 s median (4.02–4.19 s) and 2.51 GB allocated.
• This is down from 5.61–5.92 s at the previous checkpoint and about 14.5 s on the original multi-plants baseline.
• Exact manual-call lifecycle candidates fell from 59,339 with the owner-level index to 343 with the exact callee-target index.
• The final XPalm state is unchanged:
  • current_step = 4160
  • phytomer_count = 344
  • lai = 5.2085061095890355
  • ftsw = 0.7991179101191208
• The PlantBiophysics smoke profile remains stable, with no observed performance regression.

Validation and merge status

The local test-environment limitation mentioned in the previous comment has been resolved. The complete test/runtests.jl suite now passes:

• 3,887 / 3,887 tests passed
• 0 failures, 0 errors
• runtime: 1,366.4 s

The PR is mergeable and targets multi-plants. At the time of this update, no GitHub check is failing: integration, documentation, graph-editor E2E, and deployment checks are green, while six CI matrix jobs are still running.

If the remaining CI jobs complete successfully, I consider this performance tranche ready to merge into multi-plants.