diff --git a/.agents/skills/journal-build/SKILL.md b/.agents/skills/journal-build/SKILL.md index 6aba7c6f..cfdf615d 100644 --- a/.agents/skills/journal-build/SKILL.md +++ b/.agents/skills/journal-build/SKILL.md @@ -8,9 +8,10 @@ description: Build and verify Journal Apple apps for macOS, iOS Simulator and iP Run `bash .agents/skills/journal-build/scripts/build_journal_apple.sh --help` from the repository root. Maintained build, CI provisioning, icon packaging and probe-assembly helpers live in `scripts/`; product acceptance tools remain in `tool/`. The Apple entry point never installs apps, updates dependencies/pins, touches graphs or deletes caches. Coordinate CPU/actool, switch locks and borrowed inputs with active owners. - Before choosing a target or building, read [targets and prerequisites](references/targets.md). macOS requires an explicit prepared locked switch. Simulator stubs require `--link-only`; runtime imports and device builds require explicit experimental provenance. All current targets are ARM64 with deployment target >=26.0. +- For Simulator producer evidence, use `macos --producer-only --switch ...` with the same prepared inputs. It stops after native production and final input checks, returns a schema-2 envelope and `status: native-produced`, and does not build an App. Then pass that envelope to the normal explicit Simulator importer. - For `--experimental-host-restamp --native-manifest`, read [native provenance](references/native-provenance.md). Original producer records remain immutable. Host restamping does not prove full iOS cross-compilation or ABI acceptance. - For incremental reuse, switch/cache mismatch, CI changes or cold/hot measurements, read [cache and CI](references/cache-and-ci.md). Reuse existing input manifests; do not relocate switches, mix target objects, change another owner's pins, or skip locked-dependency and final source/build/test checks to obtain a hit. GitHub cache lifecycle stays in the workflow and CI dependency preparer. -Only `status: built` confirms an artifact. `preflight-passed` and `link-only` are not runtime acceptance. Preserve `cross_abi_validated=false` for host-restamped objects and hand off the manifest with the app. Installation and phone/graph operations require their own scope. +Only `status: built` confirms an App. `preflight-passed` and `link-only` are not runtime acceptance. Preserve `cross_abi_validated=false` for host-restamped objects and hand off the manifest with the app. Installation and phone/graph operations require their own scope. For build-tool changes run `PYTHONDONTWRITEBYTECODE=1 python3 .agents/skills/journal-build/scripts/test_journal_build.py`. Use `scripts/test_apple_app_icon.py` when packaging changes and the owner/environment permit actool; use `sh test/test_native_static_gmp.sh .agents/skills/journal-build/scripts/native_static_link_flags.sh` for static-link helper changes. A layout-only move needs path/import and Dune resource checks, not a new production build or GUI run. diff --git a/.agents/skills/journal-build/references/cache-and-ci.md b/.agents/skills/journal-build/references/cache-and-ci.md index 28eba1b4..b0791cbb 100644 --- a/.agents/skills/journal-build/references/cache-and-ci.md +++ b/.agents/skills/journal-build/references/cache-and-ci.md @@ -1,7 +1,9 @@ # Cache, recovery and CI measurement -Identity includes source, actual dependency ABI inputs, toolchain/SDK, flags and target. Identical inputs reuse incremental state; changed inputs select another namespace. Every run leaves external phase logs and `run-*/manifest.json`. The cache has a nonblocking lock; native Dune builds also hold the switch read lock. A busy lock belongs to its active owner: coordinate or wait. Preserve existing output/rollback apps and choose a new `--app-dir`; nothing is cleared automatically. +Identity includes source, actual dependency ABI inputs, toolchain/SDK, flags and target. Identical inputs reuse incremental state; changed inputs select another namespace. Native runs preserve a Dune process trace for actual compile/link observations. Every run leaves external phase logs and `run-*/manifest.json`. The cache has a nonblocking lock; native Dune builds also hold the switch read lock. A busy lock belongs to its active owner: coordinate or wait. Preserve existing output/rollback apps and choose a new `--app-dir`; nothing is cleared automatically. + +Producer-only and the native macOS App path share native incremental state for identical inputs. A warm producer reads the current Dune rule and revalidates the immutable evidence and live inputs before reuse; an existing cache directory alone is not a hit. Keep measurements on one source revision, explicit switch/compiler/SDK, configuration and absolute paths. For a cold sample, use a new owned external cache; never clean shared dependencies. Record actual compiler/link actions and separate phase/end-to-end elapsed time from resource or authorization waits. A failed macOS App attempt is not a successful App baseline: report its failed phase separately and compare final Simulator success only when both routes reach it. For failure, read `failed_phase`, `error` and that phase's log. Correct target/environment or select matching prepared inputs. Source/configuration/pin/current-input mismatch requires matching provenance; a historical archive change alone can use the validated receipt contract above. Missing runtime/mldoc or unresolved XMLm references require linker-rule review. Schema 2 permits an unused XMLm unit to be omitted while retaining strict locked input identity; schema 1 keeps its original force-link checks. Protected Dune/spec/lock edits still follow the repository's authorization rules. Legacy object-injecting Dune aliases remain refused; probe helpers use the isolated mode above. diff --git a/.agents/skills/journal-build/references/native-provenance.md b/.agents/skills/journal-build/references/native-provenance.md index 4a09fe7d..22e8ee99 100644 --- a/.agents/skills/journal-build/references/native-provenance.md +++ b/.agents/skills/journal-build/references/native-provenance.md @@ -14,7 +14,7 @@ compiler, pins, target C and complete-object identities retain the checks below. The link record adds an absolute `working_directory` and records the actual, expanded `-output-complete-obj` argv, including its output. Every explicit native input and its `.a`/`.o` companion must occur by absolute path in `direct_native_input_sha256`. -Both original mldoc and XMLm archives must occur in that argv with the recorded +The maintained producer also binds the runtime headers at production time, so a first target import rejects header changes after handoff. Both original mldoc and XMLm archives must occur in that argv with the recorded archive hash; response-file commands and output-as-input identities are refused. Recorded argv is inspected as data, never executed by the importer. diff --git a/.agents/skills/journal-build/references/targets.md b/.agents/skills/journal-build/references/targets.md index 946da4d8..2ea33086 100644 --- a/.agents/skills/journal-build/references/targets.md +++ b/.agents/skills/journal-build/references/targets.md @@ -27,11 +27,24 @@ bash .agents/skills/journal-build/scripts/build_journal_apple.sh ios-device --ex --sign-entitlements /path/to/device.entitlements --build-number 1.2.3 ``` +To produce native evidence without an unrelated macOS App build: + +```sh +bash .agents/skills/journal-build/scripts/build_journal_apple.sh macos --producer-only \ + --switch YOUR_SWITCH --lui-package /path/to/lui/platform/apple --cache-dir ../journal-build-cache +# Use the printed native-input.json path; keep its immutable source/link records and object. +bash .agents/skills/journal-build/scripts/build_journal_apple.sh ios-simulator \ + --experimental-host-restamp --native-manifest /printed/path/native-input.json \ + --lui-package /path/to/lui/platform/apple --cache-dir ../journal-build-cache +``` + +`--producer-only` requires native macOS inputs and rejects App output/signing modes. It keeps cache/switch locks, locked preflight, real Dune rule/input identity, original dependency bytes, runtime/mldoc and final source/compiler/dependency revalidation. It does not run macOS Swift, icons, packaging or signing. `status: native-produced` confirms only the host object and schema-2 producer evidence; it cannot be used as `validated_build` to make inputs historical. The target Simulator build still performs target C, Swift, final provider/platform checks and signing. Host restamping remains experimental with `cross_abi_validated=false`. + `--preflight` checks without compilation; remove it for the macOS build. Unset ambient compiler/OCaml/SDK lookup overrides; SDK selection is explicit. Pins come from active switch state; raw source metadata may omit `url`, while any recorded installed source must still agree. CI local pins require their matching resolution record and clean resolved commit. Provision dependencies separately with `.agents/skills/journal-build/scripts/prepare_ci_dependencies.py`, which fetches and pins; use only an authorized switch. The builder never repairs it implicitly. Offline moving refs have no remote freshness claim. ## Accept and hand off -Only `status: built` confirms an artifact. Check recorded revisions/dirty hashes, target/SDK/flags, bundle/executable hashes, version and signing identity. Resources/icons precede signing; device profile/certificate/team/entitlements and strict signature are checked. Current GMP archives are rehashed before native macOS acceptance. Experimental builds additionally check Swift object platforms, fresh target C link-map providers and runtime/mldoc retention. Record XMLm's actual retention separately from its locked input identity: schema 2 allows its unused unit to be omitted and rejects unresolved references; legacy schema 1 still requires its force-link proof and retention. Listing a `.cmxa` alone does not establish retention. +Only `status: built` confirms an App. Check recorded revisions/dirty hashes, target/SDK/flags, bundle/executable hashes, version and signing identity. Resources/icons precede signing; device profile/certificate/team/entitlements and strict signature are checked. Current GMP archives are rehashed before native macOS acceptance. Experimental builds additionally check Swift object platforms, fresh target C link-map providers and runtime/mldoc retention. Record XMLm's actual retention separately from its locked input identity: schema 2 allows its unused unit to be omitted and rejects unresolved references; legacy schema 1 still requires its force-link proof and retention. Listing a `.cmxa` alone does not establish retention. `preflight-passed` and `link-only` are not runtime acceptance. Preserve `cross_abi_validated=false` for host-restamped objects and state which compilation was reused. Carry the manifest with the App; installation and phone/graph operations are separate from build. For builder changes, use `PYTHONDONTWRITEBYTECODE=1 python3 .agents/skills/journal-build/scripts/test_journal_build.py` and the existing icon test as appropriate. diff --git a/.agents/skills/journal-build/scripts/journal_build.py b/.agents/skills/journal-build/scripts/journal_build.py index ced7adc8..529eb990 100644 --- a/.agents/skills/journal-build/scripts/journal_build.py +++ b/.agents/skills/journal-build/scripts/journal_build.py @@ -118,7 +118,8 @@ def __init__(self, args, run_dir): # They must not affect the explicit target SDK or switch in subprocesses. ignored = {name: self.env.pop(name) for name in ("SDKROOT", "CPATH", "LIBRARY_PATH", "C_INCLUDE_PATH", "CPLUS_INCLUDE_PATH") if name in self.env} - mode = ("probe-object" if getattr(args, "probe_native_object", None) else + mode = ("native-producer" if getattr(args, "producer_only", False) else + "probe-object" if getattr(args, "probe_native_object", None) else "experimental-host-restamp" if args.experimental_host_restamp else "link-only" if args.link_only else "native") self.report = {"status": "running", "started": datetime.now(timezone.utc).isoformat(), "commands": [], "mode": mode, "cross_abi_validated": False, @@ -466,7 +467,9 @@ def imported_native(self, source): raise ValueError("host object still provides DNS/crypto; target C must provide them") self.host_object, self.native_files = host, inputs self.report["native_source_sha"] = pre["commit"] - self.report["abi_limit"] = "EXPERIMENTAL: host ARM64 OCaml/runtime restamped; full iOS runtime ABI is not validated" + self.report["abi_limit"] = ("Host ARM64 producer only; no Apple App or iOS ABI validation" + if getattr(self.args, "producer_only", False) else + "EXPERIMENTAL: host ARM64 OCaml/runtime restamped; full iOS runtime ABI is not validated") print(self.report["abi_limit"], file=sys.stderr) return {"schema_version": version, "ocaml_units": units, "producer": envelope, "files": inputs, "historical_inputs": historical, @@ -679,13 +682,191 @@ def preflight(self): self.inputs = {"source": source, "swift": files(self.swift_dir), "lui": files(self.lui), "lui_sha": lui_sha, "plist": sha(self.plist), "entitlements": files(self.entitlements), "dependencies": dependencies, "toolchain": toolchain, "target": triple, - "signing": signing, "flags": {"configuration": a.configuration, "mode": self.report["mode"], + "signing": signing, "flags": {"configuration": a.configuration, + "mode": "native" if getattr(a, "producer_only", False) else self.report["mode"], "info": self.info, "sign_identity": a.sign_identity}} self.inputs["flags"]["teams"] = {name: self.env.get(name) for name in ("JOURNAL_IOS_TEAM_ID", "JOURNAL_MACOS_TEAM_ID")} self.report.update(inputs=self.inputs, target=triple, sdk_name=sdk_name, architecture="arm64", cache_key=fingerprint(self.inputs)) self.save() + def native_link_rule(self, workspace): + """Read the current rule even when Dune's incremental build ran no linker.""" + rules = json.loads(self.opam("exec", "--", "dune", "rules", "--root", ROOT, "--no-config", + "--cache=disabled", "--build-dir", workspace / "dune", "--profile", + "dev" if self.args.configuration == "debug" else "release", "--format=json", "app/native_embed.exe.o")) + if len(rules) != 1: + raise ValueError("producer needs exactly one native Dune link rule") + action, cwd = rules[0]["action"], ROOT + while isinstance(action, list) and len(action) == 3 and action[0] == "chdir": + cwd = Path(os.path.abspath(cwd / action[1])) + action = action[2] + if (not isinstance(action, list) or len(action) < 2 or action[0] != "run" + or not all(isinstance(arg, str) for arg in action[1:]) + or cwd != workspace / "dune/default"): + raise ValueError("unsupported native Dune action; expected one complete-object compiler command") + command = action[1:] + command[0] = os.path.abspath(cwd / command[0]) + compiler = self.prefix / "bin/ocamlopt.opt" + if command[0] != str(compiler) or sha(compiler) != self.inputs["toolchain"]["compiler_sha256"]: + raise ValueError("Dune producer compiler identity differs from preflight") + return cwd, command + + def producer_pin(self, name): + pins = [pin for package, pin in self.inputs["dependencies"]["pins"].items() + if package.split(".", 1)[0] == name] + if len(pins) != 1: + raise ValueError(f"producer needs one resolved {name} pin") + return pins[0] + + def producer_dependency(self, name, archive): + pin = self.producer_pin(name) + checkout, commit = Path(pin["checkout"]), pin["resolved_sha"] + tracked = self.run("git", "-C", checkout, "ls-files", "-z").split("\0") + originals = {} + for path in tracked: + if Path(path).suffix in {".ml", ".mli"}: + blob = subprocess.check_output(["git", "-C", str(checkout), "show", commit + ":" + path]) + originals.setdefault((Path(path).name, hashlib.sha256(blob).hexdigest()), []).append(path) + sources = {} + for path in sorted(archive.parent.iterdir()): + if path.suffix in {".ml", ".mli"}: + digest = sha(path) + matches = originals.get((path.name, digest), []) + if len(matches) == 1: + sources[path.name] = {"sha256": digest, "git_path": matches[0]} + if name + ".ml" not in sources: + raise ValueError(f"{name} installed source does not match its pinned Git bytes") + member = subprocess.check_output(["ar", "-p", str(archive.with_suffix(".a")), name + ".o"]) + return {"pinned_commit": commit, "source_checkout": str(checkout), "installed_archive": str(archive), + "archive_sha256": sha(archive), "static_archive_sha256": sha(archive.with_suffix(".a")), + "installed_sources_matched_to_pinned_git": sources, + "original_unit": {"cmx_sha256": sha(archive.with_suffix(".cmx")), + "object_sha256": hashlib.sha256(member).hexdigest()}} + + def produce_native(self, native, workspace): + """Produce immutable schema-2 evidence, without claiming a built Apple App.""" + self.phase = "producer-prepare" + if sha(native) != self.report["native_object"]["sha256"]: + raise ValueError("original native object hash changed before production") + self.verify_target(native, "MACOS") + cwd, original = self.native_link_rule(workspace) + directory = workspace / "native-producer" + envelope_path = directory / "native-input.json" + rule = {"working_directory": str(cwd), "command": original, + "original_object": {"path": str(native), "sha256": sha(native)}} + reused = directory.exists() + if reused: + if not envelope_path.is_file() or json.loads((directory / "dune-rule.json").read_text()) != rule: + raise ValueError("producer cache rule/object identity changed or is incomplete; choose a fresh cache") + else: + directory.mkdir() + def dump(name, value): + path = directory / name + with path.open("x") as stream: + stream.write(json.dumps(value, indent=2) + "\n") + return {"path": str(path), "sha256": sha(path)} + def absolute(arg): + return Path(os.path.abspath(cwd / arg)) + effects = [absolute(arg) for arg in original if arg.endswith("/logseq_sync_effect_runner_impl.cmxa")] + if len(effects) != 1: + raise ValueError("producer needs one original effect-runner archive") + effect = effects[0].parent + copied = directory / "host-stubs" + copied.mkdir() + inputs = {} + def record(path): + path = absolute(path) + self.bind_native_inputs(inputs, {str(path): sha(path)}) + for name in ("logseq_sync_effect_runner_impl.cmxa", "logseq_sync_effect_runner_impl.a"): + record(effect / name) + shutil.copyfile(effect / name, copied / name) + stub_name = "liblogseq_sync_effect_runner_impl_stubs.a" + patched = copied / stub_name + shutil.copyfile(effect / stub_name, patched) + if self.run("ar", "-t", patched).splitlines().count("platform_crypto_stubs.o") != 1: + raise ValueError("producer needs exactly one original host DNS/crypto C member") + self.run("ar", "-d", patched, "platform_crypto_stubs.o", env=dict(self.env, ZERO_AR_DATE="1")) + if "platform_crypto_stubs.o" in self.run("ar", "-t", patched).splitlines(): + raise ValueError("host DNS/crypto member was not removed from the owned archive copy") + command, includes = [original[0]], [] + for index, arg in enumerate(original[1:], 1): + previous = original[index - 1] + if previous == "-o": + arg = str(directory / "production-host.o") + elif previous == "-I": + path = absolute(arg) + arg = str(copied if path == effect else path) + includes.append(Path(arg)) + elif previous == "-cclib" and arg.startswith("-L") and len(arg) > 2: + arg = "-L" + str(absolute(arg[2:])) + elif previous not in {"-w", "-warn-error"} and not arg.startswith("-") and Path(arg).suffix in {".cmxa", ".cmx", ".a", ".o"}: + path = absolute(arg) + path = copied / path.name if path == effects[0] else path + arg = str(path) + record(path) + if path.suffix in {".cmxa", ".cmx"}: + record(path.with_suffix(".a" if path.suffix == ".cmxa" else ".o")) + command.append(arg) + runtime = Path(self.run(original[0], "-where")) + if runtime.resolve() != (self.prefix / "lib/ocaml").resolve(): + raise ValueError("producer compiler runtime differs from the selected switch") + # These include-search candidates bind implicit C/runtime providers as well as explicit inputs. + for include in includes + [runtime]: + for archive in sorted(include.glob("*.a")): + record(archive) + # Bind headers now: the first target import must reject later header changes, + # rather than adopting whatever bytes happen to be installed at handoff. + headers = list((self.prefix / "lib/ocaml").rglob("*.h")) + if not headers: + raise ValueError("producer OCaml runtime headers are missing") + for header in headers: + record(header) + dependencies = {} + for name in ("mldoc", "xmlm"): + archives = [Path(arg) for arg in command if Path(arg).name == name + ".cmxa"] + if len(archives) != 1: + raise ValueError(f"producer needs one original {name} archive in the actual link command") + dependencies[name] = self.producer_dependency(name, archives[0]) + pre = {"commit": self.report["source_sha"], "lui": self.lui_sha, + "signal": self.producer_pin("ocaml-signal")["resolved_sha"], + "datascript": self.producer_pin("datascript-ocaml-native")["resolved_sha"], + "native_source_sha256": {name: digest for name, digest in self.inputs["source"].items() + if Path(name).suffix in {".ml", ".mli", ".c", ".h"} + and not {"tool", "test", "apple-tests"}.intersection(Path(name).parts)}, + "installed_lui_source_sha256": {str(p): sha(p) for p in (self.prefix / "lib/lui").rglob("*") + if p.is_file() and p.suffix in {".ml", ".mli"}}, + "production_native_object": str(directory / "production-host.o")} + link = {"command": command, "working_directory": str(cwd), "direct_native_input_sha256": inputs, + "compiler_sha256": self.inputs["toolchain"]["compiler_sha256"], + "compiler_version": self.run(original[0], "-version"), "new_dependency_provenance": dependencies, + "patched_host_stub_archive_sha256": sha(patched), + "device_c_source_sha256": {p.name: sha(p) for p in + (ROOT / "logseq_sync/lib/effect_runner/platform").glob("platform_crypto_stubs.*")}} + # Reject unbound/unknown command forms before executing the current Dune-derived relink. + self.ordinary_native_inputs(pre, link) + link["command"] = self.ordinary_command + self.phase = "producer-link" + self.run(*link["command"], cwd=cwd) + if any(sha(path) != digest for path, digest in inputs.items()): + raise ValueError("native link input hash changed during production") + pre["production_native_object_sha256"] = link["production_host_sha256"] = sha(pre["production_native_object"]) + envelope = {"schema_version": 2, "source": dump("source.json", pre), "link": dump("link.json", link), + "dune_rule": dump("dune-rule.json", rule)} + dump("native-input.json", envelope) + self.phase = "verify-producer" + previous = self.args.native_manifest + try: + self.args.native_manifest = envelope_path + imported = self.imported_native(self.inputs["source"]) + finally: + self.args.native_manifest = previous + self.native_files[str(native)] = rule["original_object"]["sha256"] + self.report["producer"] = {"envelope": {"path": str(envelope_path), "sha256": sha(envelope_path)}, + "object": {"path": str(self.host_object), "sha256": sha(self.host_object)}, + "ocaml_units": imported["ocaml_units"], "link_executed": not reused} + return envelope_path + def compile(self, workspace): a = self.args self.phase = "native" @@ -714,7 +895,7 @@ def compile(self, workspace): self.env.update(DUNE_CACHE_ROOT=str(workspace / "dune-cache"), XDG_CACHE_HOME=str(workspace / "xdg-cache")) self.opam("exec", "--", "dune", "build", "--root", ROOT, "--no-config", "--cache=disabled", - "--build-dir", dune_dir, + "--build-dir", dune_dir, "--trace-file", self.run_dir / "native-trace.json", "--profile", "dev" if a.configuration == "debug" else "release", "-j", str(a.jobs), "app/native_embed.exe.o") native = dune_dir / "default/app/native_embed.exe.o" @@ -724,6 +905,8 @@ def compile(self, workspace): self.report["native_object"] = {"path": str(native), "sha256": sha(native), "mode": self.report["mode"]} if native_units is not None: self.report["native_object"]["ocaml_units"] = native_units + if getattr(a, "producer_only", False): + return self.produce_native(native, workspace) self.phase = "swift" scratch = workspace / "swift" native_inputs = [native, self.target_c] if a.native_manifest else [native] @@ -904,6 +1087,10 @@ def verify_inputs(self): raise RuntimeError("imported native/signing inputs changed during build; artifact is not accepted") a = self.args if not (a.link_only or a.native_manifest or getattr(a, "probe_native_object", None)): + if getattr(a, "producer_only", False): + current = opam_identity(self.opam) + if any(self.inputs["toolchain"].get(name) != value for name, value in current.items()): + raise RuntimeError("OCaml compiler changed during build; producer is not accepted") if self.dependencies() != self.inputs["dependencies"]: raise RuntimeError("switch changed during build; artifact is not accepted") @@ -916,6 +1103,7 @@ def arguments(): parser.add_argument("--switch", help="explicit pre-provisioned opam switch (native macOS only)") parser.add_argument("--lui-package", type=Path, help="local LUI platform/apple package") parser.add_argument("--preflight", action="store_true", help="check inputs without compiling") + parser.add_argument("--producer-only", action="store_true", help="native macOS producer evidence only; no Swift, App packaging or signing") parser.add_argument("--link-only", action="store_true", help="explicit stub build; not runnable acceptance") parser.add_argument("--experimental-host-restamp", action="store_true", help="explicit existing host ARM64 compatibility route; full iOS ABI unverified") parser.add_argument("--native-manifest", type=Path, help="hash-verified source/link/XMLm producer evidence envelope") @@ -930,6 +1118,10 @@ def arguments(): parser.add_argument("--provisioning-profile", type=Path, help="existing device profile; never downloaded") parser.add_argument("--sign-entitlements", type=Path, help="existing device signing entitlements") a = parser.parse_args() + if a.producer_only and (a.platform != "macos" or a.link_only or a.native_manifest or a.experimental_host_restamp + or a.probe_native_object or a.app_dir or a.app_directory or a.sign_identity != "-" + or a.provisioning_profile or a.sign_entitlements): + parser.error("producer-only requires native macOS --switch inputs without App output/import/signing modes") if a.probe_native_object and (a.platform == "ios-device" or a.switch or a.link_only or a.native_manifest or a.experimental_host_restamp): parser.error("probe mode is isolated macOS/Simulator object import, separate from production/stub modes") for name in ("JOURNAL_OCAML_OBJECT", "JOURNAL_EXTRA_OBJECTS", "JOURNAL_NATIVE_LINK_INPUTS"): @@ -995,12 +1187,13 @@ def interrupted(signum, frame): build.report["cache_reused"] = (workspace / "inputs.json").exists() (workspace / "inputs.json").write_text(json.dumps(build.inputs, indent=2) + "\n") product = build.compile(workspace) - build.assemble(product, a.app_dir or run_dir / "LogseqJournal.app") + if not a.producer_only: + build.assemble(product, a.app_dir or run_dir / "LogseqJournal.app") build.verify_inputs() - build.report["status"] = "preflight-passed" if a.preflight else "built" + build.report["status"] = "preflight-passed" if a.preflight else "native-produced" if a.producer_only else "built" build.report["finished"] = datetime.now(timezone.utc).isoformat() build.save() - print(build.report.get("artifact", {}).get("app", "preflight passed")) + print(build.report.get("artifact", {}).get("app", "preflight passed") if a.preflight or not a.producer_only else product) print(f"manifest: {run_dir / 'manifest.json'}", file=sys.stderr) except (RuntimeError, OSError, ValueError, KeyError, TypeError, subprocess.SubprocessError, KeyboardInterrupt) as error: build.report.update(status="failed", failed_phase=build.phase, error=str(error)) diff --git a/.agents/skills/journal-build/scripts/test_journal_build.py b/.agents/skills/journal-build/scripts/test_journal_build.py index b2e91dc1..7c45b1cc 100644 --- a/.agents/skills/journal-build/scripts/test_journal_build.py +++ b/.agents/skills/journal-build/scripts/test_journal_build.py @@ -105,6 +105,22 @@ def test_existing_output_is_preserved_before_toolchain_checks(self): self.assertIn("exists", result.stderr) self.assertEqual(sentinel.read_text(), "keep") + def test_producer_only_is_an_explicit_native_macos_mode(self): + for args in [('ios-simulator', '--link-only'), + ('macos', '--link-only'), + ('macos', '--probe-native-object', '/tmp/probe.o'), + ('macos', '--experimental-host-restamp', '--native-manifest', '/tmp/input.json'), + ('macos', '--switch', 'unused', '--app-dir', '/tmp/unused-producer.app'), + ('macos', '--switch', 'unused', '--sign-identity', 'unused')]: + with self.subTest(args=args): + result = self.cli(*args, '--producer-only', '--preflight') + self.assertNotEqual(result.returncode, 0) + self.assertIn('producer-only requires', result.stderr) + result = self.cli('macos', '--producer-only', '--preflight') + self.assertNotEqual(result.returncode, 0) + self.assertIn('--switch', result.stderr) + + class CacheBoundaryTests(unittest.TestCase): @classmethod @@ -529,6 +545,25 @@ def test_final_verification_rejects_changed_gmp_and_probe_input(self): with self.assertRaisesRegex(RuntimeError, 'native'): build.verify_inputs() + def test_producer_final_verification_rechecks_the_selected_compiler(self): + with self.pinned_switch() as f: + build = f.build + build.args.producer_only = True + compiler = self.compiler_fixture(build.prefix) + build.swift_dir, build.lui, build.entitlements = (f.root / name for name in ('swift', 'lui', 'entitlements')) + for path in (build.swift_dir, build.lui, build.entitlements): + path.mkdir() + build.plist = f.root / 'Info.plist' + build.plist.write_bytes(b'plist fixture') + build.tools = {} + build.inputs = dict(source=self.build.source_inputs(), swift={}, lui={}, entitlements={}, + plist=self.build.sha(build.plist), signing={}, dependencies=build.dependencies(), + toolchain=dict(hashes={}, sdk_identity={'settings': {}}, **self.build.opam_identity(build.opam))) + build.verify_inputs() + compiler.write_text(compiler.read_text() + '# compiler changed during native phase\n') + with self.assertRaisesRegex(RuntimeError, 'compiler|toolchain'): + build.verify_inputs() + def test_probe_helper_preflight_supports_stub_and_platform_matched_objects(self): sys.path.insert(0, str(SCRIPTS)) import lui_probe_host @@ -900,6 +935,197 @@ def save(): yield SimpleNamespace(build=build, source=source, link=link, pre=pre, native=native, envelope=envelope, path=path, root=root, save=save, runtime=runtime, provenance=provenance) + @contextmanager + def native_dune_producer(self): + """A real Dune complete-object rule, including removable host C providers.""" + with self.ordinary_producer_fixture() as f: + repo = self.build.ROOT + original_compiler = Path(f.link['command'][0]) + compiler = f.root / 'package/bin/ocamlopt.opt' + compiler.parent.mkdir(exist_ok=True) + compiler.symlink_to(original_compiler) + (f.root / 'package/lib/ocaml').symlink_to(f.runtime) + switch = os.environ.get('JOURNAL_TEST_SWITCH') + dune = (Path(subprocess.check_output(['opam', 'var', '--safe', '--switch', switch, 'bin'], + text=True).strip()) / 'dune') if switch else Path(shutil.which('dune')) + # Mirror opam exec's selected compiler PATH; omit ambient OCaml fallback even locally. + compiler_bin = dune.parent if switch else original_compiler.parent + f.build.env['PATH'] = str(compiler_bin) + os.pathsep + os.defpath + effect = repo / 'logseq_sync/lib/effect_runner' + effect.mkdir(exist_ok=True) + original = f.root / 'effect' + for path in original.glob('*'): + if path.is_file(): + shutil.copyfile(path, effect / path.name) + c = effect / 'platform_crypto_stubs.c' + c.write_text('int logseq_transport_dns_start(void) { return 1; }\n' + 'int logseq_journal_crypto_call(void) { return 1; }\n') + obj = c.with_suffix('.o') + subprocess.run(['xcrun', 'clang', '-c', c, '-o', obj], check=True, capture_output=True) + stub = effect / 'liblogseq_sync_effect_runner_impl_stubs.a' + subprocess.run(['ar', 'rcs', stub, obj], check=True, capture_output=True) + archive = effect / 'logseq_sync_effect_runner_impl.cmxa' + subprocess.run([compiler, '-a', '-o', archive, effect / 'effect.cmx', + '-cclib', '-llogseq_sync_effect_runner_impl_stubs'], check=True, capture_output=True) + # The fixture rule is opaque to the builder: it must read actual Dune argv. + command = [str(compiler), '-output-complete-obj', '-w', '@1..3', '-o', 'app/native_embed.exe.o', + '-I', 'logseq_sync/lib/effect_runner', 'logseq_sync/lib/effect_runner/' + archive.name, + f.provenance['xmlm']['installed_archive'], f.provenance['mldoc']['installed_archive'], + 'app/unit.cmx'] + def atom(value): + return json.dumps(str(value)) + (repo / 'dune-project').write_text('(lang dune 3.0)\n') + (repo / 'app/dune').write_text('(rule (target native_embed.exe.o) (deps libgmp.a unit.cmx unit.o ' + + ' '.join(atom('../logseq_sync/lib/effect_runner/' + path.name) + for path in effect.iterdir() if path.suffix in ('.cmxa', '.a')) + + ') (action (chdir .. (run ' + ' '.join(map(atom, command)) + '))))\n') + shutil.copyfile(stub, repo / 'app/libgmp.a') + workspace = f.root / 'workspace' + workspace.mkdir() + dune_dir = workspace / 'dune' + def opam(*argv): + self.assertEqual(argv[:3], ('exec', '--', 'dune')) + return f.build.run(dune, *argv[3:]) + f.build.opam = opam + f.build.args = SimpleNamespace(platform='macos', configuration='release', jobs=1, + producer_only=True, link_only=False, experimental_host_restamp=False, + native_manifest=None, probe_native_object=None) + f.build.report['mode'] = 'native-producer' + f.build.tools = {'swift': '/unexpected-swift-command'} + f.build.prefix = f.root / 'package' + subprocess.run(['git', '-C', repo, 'add', '.'], check=True, capture_output=True) + subprocess.run(['git', '-C', repo, '-c', 'user.name=Build test', '-c', 'user.email=build@example.invalid', + 'commit', '-qm', 'native rule'], check=True, capture_output=True) + f.build.report['source_sha'] = subprocess.check_output(['git', '-C', repo, 'rev-parse', 'HEAD'], text=True).strip() + f.build.inputs = dict(source=self.build.source_inputs(), lui_sha=f.pre['lui'], + toolchain=dict(compiler_sha256=self.build.sha(compiler), + opam_prefix=str(f.build.prefix)), + dependencies=dict(pins={name + ('.0.1.0' if name == 'ocaml-signal' else '.dev'): dict(resolved_sha=commit, + checkout=f.provenance[name]['source_checkout'] if name in f.provenance else str(repo)) + for name, commit in [('mldoc', f.provenance['mldoc']['pinned_commit']), + ('xmlm', f.provenance['xmlm']['pinned_commit']), + ('ocaml-signal', f.pre['signal']), + ('datascript-ocaml-native', f.pre['datascript'])]})) + f.build.opam('exec', '--', 'dune', 'build', '--root', repo, '--no-config', + '--cache=disabled', '--build-dir', dune_dir, '--profile', 'release', + '-j', '1', 'app/native_embed.exe.o') + native = dune_dir / 'default/app/native_embed.exe.o' + f.build.report['native_object'] = dict(path=str(native), sha256=self.build.sha(native), mode='native-producer') + yield SimpleNamespace(**vars(f), workspace=workspace, original_native=native, compiler=compiler) + + def test_producer_uses_current_dune_rule_even_after_a_noop_build(self): + self.assertTrue(hasattr(self.build.Build, 'produce_native'), 'producer-only entry is missing') + with self.native_dune_producer() as f: + for attempt in range(2): + if attempt: + f.build.opam('exec', '--', 'dune', 'build', '--root', self.build.ROOT, '--no-config', + '--cache=disabled', '--build-dir', f.workspace / 'dune', '--profile', 'release', + '-j', '1', 'app/native_embed.exe.o') + envelope_path = f.build.compile(f.workspace) + envelope = json.loads(envelope_path.read_text()) + self.assertEqual(envelope['schema_version'], 2) + self.assertNotIn('validated_build', envelope) + source = self.build.read_evidence(envelope['source'], envelope_path.parent) + link = self.build.read_evidence(envelope['link'], envelope_path.parent) + self.assertEqual(link['compiler_sha256'], self.build.sha(f.compiler)) + self.assertEqual(self.build.sha(f.original_native), f.build.report['native_object']['sha256']) + self.assertNotEqual(source['production_native_object'], str(f.original_native)) + f.build.args.native_manifest = envelope_path + imported = f.build.imported_native(f.build.inputs['source']) + f.build.args.native_manifest = None + self.assertEqual(imported['ocaml_units'], dict(runtime=True, mldoc=True, xmlm=False)) + inputs = imported['files'] + for name, digest in link['direct_native_input_sha256'].items(): + self.assertEqual(inputs[name], digest) + self.assertNotIn('artifact', f.build.report) + self.assertFalse(f.build.report['cross_abi_validated']) + self.assertFalse(f.build.report['runtime_acceptance']) + self.assertEqual(f.build.report['producer']['envelope']['sha256'], self.build.sha(envelope_path)) + if not attempt: + first_path = envelope_path + first_bytes = envelope_path.read_bytes() + first_mtime = envelope_path.stat().st_mtime_ns + self.assertEqual(first_path, envelope_path) + self.assertEqual(first_bytes, envelope_path.read_bytes()) + self.assertEqual(first_mtime, envelope_path.stat().st_mtime_ns, 'immutable producer records were overwritten') + links = [record for record in f.build.report['commands'] if record['phase'] == 'producer-link' + and record['argv'][0] == str(f.compiler)] + self.assertEqual(len(links), 1, 'hot producer unexpectedly linked again') + + def test_producer_binds_runtime_headers_before_target_handoff(self): + with self.native_dune_producer() as f: + envelope = f.build.produce_native(f.original_native, f.workspace) + headers = f.build.prefix / 'lib/ocaml' + original = {path.relative_to(headers): path.read_bytes() for path in headers.rglob('*.h')} + self.assertIn(Path('caml/memory.h'), original) + # Replace only the fixture's symlink with private header copies. + headers.unlink() + for relative, data in original.items(): + path = headers / relative + path.parent.mkdir(parents=True, exist_ok=True) + path.write_bytes(data) + changed = headers / 'caml/memory.h' + changed.write_bytes(changed.read_bytes() + b'\n/* changed after producer handoff */\n') + f.build.args.native_manifest = envelope + with self.assertRaisesRegex(ValueError, 'header|hash|conflicting'): + f.build.imported_native(f.build.inputs['source']) + + def test_producer_reuses_native_bytes_after_a_documentation_commit(self): + self.assertTrue(hasattr(self.build.Build, 'produce_native'), 'producer-only entry is missing') + with self.native_dune_producer() as f: + envelope = f.build.produce_native(f.original_native, f.workspace) + doc = self.build.ROOT / 'README.md' + doc.write_text('documentation-only successor\n') + subprocess.run(['git', '-C', self.build.ROOT, 'add', doc], check=True, capture_output=True) + subprocess.run(['git', '-C', self.build.ROOT, '-c', 'user.name=Build test', + '-c', 'user.email=build@example.invalid', 'commit', '-qm', 'documentation'], + check=True, capture_output=True) + f.build.report['source_sha'] = subprocess.check_output( + ['git', '-C', self.build.ROOT, 'rev-parse', 'HEAD'], text=True).strip() + self.assertEqual(f.build.inputs['source'], self.build.source_inputs()) + try: + reused = f.build.produce_native(f.original_native, f.workspace) + except ValueError as error: + self.fail(f'documentation-only commit rejected identical native inputs: {error}') + self.assertEqual(reused, envelope) + self.assertFalse(f.build.report['producer']['link_executed']) + + def test_producer_refuses_a_rule_with_unbound_linker_inputs_before_running_it(self): + self.assertTrue(hasattr(self.build.Build, 'produce_native'), 'producer-only entry is missing') + with self.native_dune_producer() as f: + path = self.build.ROOT / 'app/dune' + path.write_text(path.read_text().replace('"-w"', '"-ccopt" "@unbound.rsp" "-w"')) + f.build.inputs['source'] = self.build.source_inputs() + with self.assertRaisesRegex(ValueError, 'unsupported|native.*argument'): + f.build.produce_native(f.original_native, f.workspace) + self.assertNotIn('producer', f.build.report) + + def test_producer_reuse_rechecks_original_object_and_immutable_records(self): + self.assertTrue(hasattr(self.build.Build, 'produce_native'), 'producer-only entry is missing') + for changed in ('original-object', 'evidence'): + with self.subTest(changed=changed), self.native_dune_producer() as f: + envelope = f.build.produce_native(f.original_native, f.workspace) + if changed == 'original-object': + f.original_native.chmod(f.original_native.stat().st_mode | 0o200) + f.original_native.write_bytes(f.original_native.read_bytes() + b'changed') + else: + record = json.loads(envelope.read_text())['link'] + Path(record['path']).write_text('{}') + with self.assertRaisesRegex((RuntimeError, ValueError), 'hash|changed|identity'): + f.build.produce_native(f.original_native, f.workspace) + + def test_producer_only_receipt_cannot_authorize_historical_target_inputs(self): + with self.ordinary_producer_fixture() as f: + imported = f.build.imported_native(f.source) + receipt = dict(status='native-produced', mode='native-producer', source_sha=f.pre['commit'], + inputs=dict(dependencies=imported), production_link=dict(ocaml_units=imported['ocaml_units'])) + receipt_path = f.root / 'producer-receipt.json' + receipt_path.write_text(json.dumps(receipt)) + f.envelope['validated_build'] = dict(path=str(receipt_path), sha256=self.build.sha(receipt_path)) + f.save() + with self.assertRaisesRegex(ValueError, 'receipt'): + f.build.imported_native(f.source) + def test_ordinary_xmlm_input_allows_dead_code_elimination(self): with self.ordinary_producer_fixture() as f: imported = f.build.imported_native(f.source) diff --git a/docs/agent-guide/implemented/process/2026-10-17-journal-native-producer.md b/docs/agent-guide/implemented/process/2026-10-17-journal-native-producer.md new file mode 100644 index 00000000..7c511e2c --- /dev/null +++ b/docs/agent-guide/implemented/process/2026-10-17-journal-native-producer.md @@ -0,0 +1,58 @@ +# Journal Native Producer + +## Problem + +Simulator host-restamp evidence currently requires the native Dune phase of a macOS App build, which then unconditionally runs unrelated macOS Swift, packaging and signing. At PR90 head 63fce61 the native phase succeeds but macOS Swift fails on JournalChrome listSectionSpacing. The external producer script can consume that phase but the failed App manifest is not successful validation. + +## Proposal + +The user explicitly authorized a minimal producer-only entry, accurate receipts, controlled timing and a PR. Add macOS `--producer-only` to the existing Apple builder: retain prepared-switch/cache locks and complete preflight, build native_embed, derive schema-2 producer inputs from the current Dune rule, remove host DNS/crypto only from an owned archive copy, link the fixed host object, validate it through the existing importer, bind runtime headers before target handoff, and recheck all live inputs before accepting `status: native-produced`. Return its native-input envelope. Do not run macOS Swift/package/sign or grant historical-input privileges from this receipt. Simulator uses the ordinary explicit experimental importer and retains target C/Swift/provider/signing checks. + +The build tools own this boundary; pure application reducer events cannot run Dune, observe filesystem input mutation or represent build phase completion. Tests belong in the existing build-owner Python suite, with a tiny real compiler/Dune fixture and CLI rejection checks. + +## Decision + +Implement the user-authorized producer-only scope in the existing builder. Keep the full App default path and schema-1/2 import contracts, use native-produced only for producer acceptance, and require the ordinary experimental target build for an App. No protected spec/Dune/pin change is needed. + +## Alternatives considered + +### Keep full macOS App as prerequisite + +Rejected: it couples producer success to unrelated platform Swift compatibility. A failed App is not a successful end-to-end timing baseline. + +### Weaken import or reuse a failed receipt + +Rejected: keep PR90 schema-2 identity, runtime/mldoc retention, XMLm unresolved-reference and complete target acceptance checks. Do not alter Dune/spec/pins or another owner workspace. + +## Acceptance criteria + +- Producer-only uses one prepared locked switch and external cache; no unrelated macOS App stages run and receipt cannot claim App/runtime/iOS ABI acceptance. +- Real Dune link argv, compiler, source/config/pins and original dependency/archive/unit bytes remain bound and rechecked; hot runs obtain a current rule rather than guessing from old trace or cache existence. +- A normal Simulator experimental build consumes the generated schema-2 envelope and passes the existing final checks. Full source build/tests and independent journal-code-review bind the final head. +- Controlled measurements preserve source/lock/compiler/SDK/config/path and cache state, distinguish cold/hot, phase/end-to-end and queue/approval waits, and report failed old workflow boundaries honestly. + +## Risks + +- Host ARM64 restamping remains experimental with cross_abi_validated=false; this producer proves neither a macOS App nor full OCaml iOS ABI. +- New source or dependency bytes require fresh evidence. Producer receipts keep consumed files live until a successful target receipt validates them. +- PR90 is unmerged; the independent branch now stacks on its pushed, reviewed and CI-green fec63d8 commit. Do not merge the parent PR. + +## Consequences + +Simulator producers no longer depend on unrelated macOS Swift compatibility. Identity, live input/header checks and target acceptance remain mandatory; producer evidence alone cannot grant historical-input privileges or iOS ABI acceptance. Timing evidence is bounded to the measured failed auxiliary Swift stage and qualified route totals above the unchanged native/target checks. + +## Validation + +All 54 build-owner tests passed locally on the measured material-code snapshot in 67.449s with no skips. The real Dune/OCaml fixture proves fresh production, hot reuse without relinking, immutable records, unknown-argument refusal, ancestor reuse, and header mutation rejection before target handoff. Header rejection and documentation-only reuse were observed RED before correction. Independent read-only journal-code-review at 801cddf closed both findings and found no further actionable issue; final whole-source build/regression remains the exact-head PR CI gate. + +The first exact-head CI then exposed a fixture-only PATH dependency: restored cache/probes/locked dependencies succeeded, but the real Dune fixture could not find globally exposed ocamlc. The fixture now sets only the selected compiler bin plus system tool paths. A stripped-PATH reproduction failed first, then two real cases passed in 8.803s with the isolated environment. Production builder bytes and product source remain identical to the measurement snapshot; the full suite and source regression run on the updated exact-head CI. + +One bounded cold/hot pair ran on source/native 801cddf, using one prepared switch/root, compiler, locks, SDKs, Release/jobs=2 and the same absolute native cache path. New owned caches were renamed/restored in place; no shared dependency cache was cleaned. Real Dune traces show 268 OCaml compilation calls, 11 archive calls, one complete-object link and three C compilations in each cold sample; both hot samples show zero compilation/archive/link actions. Producer relinking runs once cold and zero times hot. All four final Simulator builds passed current source/input, target C provider, 54 Swift object platform, runtime/mldoc and signature checks; XMLm was accurately absent. They retain cross_abi_validated=false and runtime_acceptance=false. + +The removed macOS Swift subprocess cost 35.273704s cold and 2.694505s hot. The legacy auxiliary macOS App failed at listSectionSpacing; packaging/signing were not executed and have no measured saving. New producer-only CLI elapsed was 24.124s cold and 6.874s hot; its following Simulator build was 45.923s/6.111s. Observed complete recovery-route totals were legacy 112.731s/18.944s versus producer-only 70.053s/12.991s. These totals are not a strict successful-App acceleration claim: the legacy App failed and 14 independently regenerated archive hashes differed only in date metadata (all member payloads identical), changing the derived Simulator namespace. The successful old macOS App baseline and exact equivalent end-to-end saving are therefore not established. Cold means a fresh owned Apple/Dune/Swift cache, not cold OS caches or a dependency reinstall. Resource, authorization and preflight setup waits were excluded from the formal execution measurements. + +Raw argv, sanitized actual process events, phase logs, native/envelope/receipt/product hashes and archive metadata comparisons remain outside the repository in this task's evidence directory. No repeated benchmark or unrelated macOS UI repair was performed. + +## Questions + +None. Scope, protected-file exclusions and measurement constraints were explicitly authorized in the delegated user request; routine implementation choices remain within that scope. Formal timing completed in the coordinated exclusive Mac window; its bounded results and limitations are recorded above.