diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 7d16253b..090ddbf9 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -114,10 +114,10 @@ jobs: run: sbt +update - name: Check formatting - run: sbt '++ ${{ matrix.scala }}' scalafmtCheckAll scalafmtSbtCheck benchmarks/scalafmtCheck demoUplcJvm/scalafmtCheckAll demoUplcJvmValidators/scalafmtCheckAll demoUplcJvmBench/scalafmtCheckAll + run: sbt '++ ${{ matrix.scala }}' scalafmtCheckAll scalafmtSbtCheck benchmarks/scalafmtCheck demoUplcJvm/scalafmtCheckAll demoUplcJvmValidators/scalafmtCheckAll demoUplcJvmBench/scalafmtCheckAll demoGrpcFilter/scalafmtCheckAll demoGrpcFilterBench/scalafmtCheckAll - name: Check scalafix - run: sbt '++ ${{ matrix.scala }}' 'scalafixAll --check' 'demoUplcJvm/scalafixAll --check' 'demoUplcJvmValidators/scalafixAll --check' 'demoUplcJvmBench/scalafixAll --check' + run: sbt '++ ${{ matrix.scala }}' 'scalafixAll --check' 'demoUplcJvm/scalafixAll --check' 'demoUplcJvmValidators/scalafixAll --check' 'demoUplcJvmBench/scalafixAll --check' 'demoGrpcFilter/scalafixAll --check' 'demoGrpcFilterBench/scalafixAll --check' - name: Check that workflows are up to date run: sbt githubWorkflowCheck @@ -136,13 +136,16 @@ jobs: - name: Test UPLC → JVM demo (differential vs Scalus CEK) run: sbt '++ ${{ matrix.scala }}' demoUplcJvm/test demoUplcJvmBench/Jmh/compile + - name: Test gRPC filter demo (eo-jsoniter + fs2) + run: sbt '++ ${{ matrix.scala }}' demoGrpcFilter/test demoGrpcFilterBench/Jmh/compile + - name: Make target directories if: github.event_name != 'pull_request' && (startsWith(github.ref, 'refs/tags/v')) - run: mkdir -p jsoniter/target benchmarks/target kyo/target target demos/uplc-jvm-bench/target circe/target zio/target unidocs/target avro/target laws/target tests/target generics/target demos/uplc-jvm-validators/target schemes-laws/target schemes/target core/target demos/uplc-jvm/target project/target + run: mkdir -p demos/grpc-filter/target jsoniter/target benchmarks/target kyo/target target demos/uplc-jvm-bench/target circe/target zio/target unidocs/target avro/target laws/target tests/target generics/target demos/uplc-jvm-validators/target schemes-laws/target schemes/target core/target demos/uplc-jvm/target demos/grpc-filter-bench/target project/target - name: Compress target directories if: github.event_name != 'pull_request' && (startsWith(github.ref, 'refs/tags/v')) - run: tar cf targets.tar jsoniter/target benchmarks/target kyo/target target demos/uplc-jvm-bench/target circe/target zio/target unidocs/target avro/target laws/target tests/target generics/target demos/uplc-jvm-validators/target schemes-laws/target schemes/target core/target demos/uplc-jvm/target project/target + run: tar cf targets.tar demos/grpc-filter/target jsoniter/target benchmarks/target kyo/target target demos/uplc-jvm-bench/target circe/target zio/target unidocs/target avro/target laws/target tests/target generics/target demos/uplc-jvm-validators/target schemes-laws/target schemes/target core/target demos/uplc-jvm/target demos/grpc-filter-bench/target project/target - name: Upload target directories if: github.event_name != 'pull_request' && (startsWith(github.ref, 'refs/tags/v')) diff --git a/CLAUDE.md b/CLAUDE.md index 6ad4de29..cf4aaa60 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -30,12 +30,14 @@ Test-only: `org.typelevel:discipline-specs2_3:2.0.0`. | `benchmarks` | `benchmarks/` | — (not published) | JMH benchmarks vs Monocle (not part of root aggregate) | | `demoUplcJvm` | `demos/uplc-jvm/` | — (not published) | Demo: UPLC (Cardano Plutus Core) → JVM bytecode as a chain of LAWFUL monomorphic eo optics, `CborScript ⇀ TypedProgram ≅ JvmClass ⇀ JvmClassScript` (stages `toTyped`, `lowering`, `compiling` — the last is `UplcJvm.compile` / `decompile`; callers use `UplcJvm.toTyped` / `toJvmClass` / `toBytecode`, all from the script, all `MendTearPrism`s). Two PRISMS around one iso: Scalus decodes ANY declared Plutus Core version, but the ledger takes only those Plutus V3 has at PV11 (`Semantics.versions` = 1.0.0, 1.1.0), so `toTyped` misses on any other; a class can exceed the JVM's 64 KB method / 65 535-constant limits, so `compiling` misses (`UplcJvm.compile` gives the limit as a `Left` — the one catch of ASM's `MethodTooLargeException` / `ClassTooLargeException`). `lowering`'s way back types the embedded script WITHOUT the version check (only a typed program is lowered). Lawful because every stage keeps its script — `TypedProgram` its source, the class the script as Base64 constants — so `reverseGet` walks back exactly; `OpticSpec` runs cats-eo-laws' `PrismTests` on `toTyped` and `toBytecode` (JvmClass has no equality, so no `IsoTests` for `lowering`); tests reach stages through `Oracle.typed` / `jvmClass` / `classFile`, and `Oracle.compare` checks the round trips on every program. The four types hand out COPIES only (`bytes`, `tree`), so nothing changes a value after the compiler made or checked it. Compile time is STACK SAFE (`OpticSpec` compiles 100 000-deep programs on a 1 MB stack): decode, `Typer` and `Lower` are walks of `Step`s (`Step.scala`) — plain calls while shallow, `cats.Eval` on the heap from `Walk.Deep` (128) calls deep — so EVERY recursive call must go through `Walk.nested`. Trampolining every node (plain `Eval`) made translation ~2× slower. Steps run once each, in order, so `Lower` writes instructions as a step is made. Never sequence effectful `Eval`s with cats' `traverse_` / `void`: cats treats `Eval` as pure and drops a step whose result is unused. Scalus' flat decoder/encoder still recurse (overflow at 2 000–20 000 deep on 1 MB). `CborScript` / `JvmClassScript` are opaque over Scalus' `ByteString` (value equality, so the laws can use `==`), `JvmClass` opaque over ASM's `ClassNode` (made only by `Lower` and `UplcJvm.decompile`; `.tree` to inspect); `CborScript.parse` is the checked entry for untrusted bytes. Compiled with `-opt -opt-inline:**,!scala.**`: inlining from the Scala library made the short validations 12–27% slower (it grows `Rt.force` past what the JIT inlines), everything else measured neutral to ~5% faster. Keep generated code free of unreachable instructions (the typer drops what follows a failure): dead code leaves constant-pool entries that break the class file's read/write fixpoint. TypedProgram = typed Scala-like expression over Scalus' own `Term`s (hybrid typing: types are Scalus `DefaultUni`s, builtin signatures its `TypeScheme`s, instantiated at argument types; unboxed `BigInt`/`boolean`/`ByteString`/`String` where provable, boxed Scalus `Constant`s elsewhere), JvmClass written by asm-commons' `InstructionAdapter` (no own IR — and not a library above ASM: those cost translation speed), class file by ASM `ClassWriter` (computes frames). Saturated builtins are direct JVM calls of `scalus.uplc.builtin.Builtins` (or lazy-`Data` natives in `Rt`); the rest, and builtins used as values, run Scalus' CEK meanings. Reuse Scalus code before adding demo-local code: failures are `MachineError`, `Exec.run` returns `eval.Result`. Correctness is DIFFERENTIAL against the Scalus 1.3.0 CEK machine (Plutus V3 @ PV11, variant E — `Semantics.scala`): vendored Plutus conformance suite (`src/test/resources/plutus-conformance`, regenerate with `scripts/refresh-conformance.sh`, never hand-edit) + validators. Result/error only — no budget metering. Not in the root aggregate (Scalus pulls JNI crypto); CI runs `demoUplcJvm/test` as its own step. See its README | | `demoUplcJvmValidators` | `demos/uplc-jvm-validators/` | — (not published) | On-chain validators compiled to UPLC by `scalus-plugin` (isolated so the plugin never sees the compiler's sources), plus the shared off-chain fixtures (`Contexts`, `Scenarios`); test-scope dep of `demoUplcJvm` | -| `demoUplcJvmBench` | `demos/uplc-jvm-bench/` | — (not published) | JMH for the demo: CEK machine vs generated class per validator, Fibonacci scaling, translation cost. NOT part of `benchmarks/`, the nightly sweep or `BENCHMARKS.md`; its numbers live on `site/docs/demos/uplc-jvm.md`. CI only compiles it (`demoUplcJvmBench/Jmh/compile`) | +| `demoUplcJvmBench` | `demos/uplc-jvm-bench/` | — (not published) | JMH for the demo: CEK machine vs generated class per validator, Fibonacci scaling, translation cost. NOT part of `benchmarks/`, the nightly sweep or `BENCHMARKS.md`; its numbers live on `site/docs/cookbook/uplc-jvm.md` (the cookbook's "Blockchain smart contract compilation demo" page). CI only compiles it (`demoUplcJvmBench/Jmh/compile`) | +| `demoGrpcFilter` | `demos/grpc-filter/` | — (not published) | Demo: an fs2 filter on a gRPC stream read as raw bytes off a TCP socket — Length-Prefixed-Message frames (`GrpcFrames.splitFrames` / `joinFrames` pipes) of JSON `Issue`s (64 fields; the 61st, `detail`, a 3-case enum nested 3 objects deep). It classifies and filters every message with eo-jsoniter prisms WITHOUT decoding it (`Wire.detail` = `JsoniterPrism[Issue].field(_.detail).raw` slices the detail's bytes into the opaque `DetailJson`; typed per-case leaf prisms, `JsoniterPrism[Detail.Incident].field(_.impact)…`, read it) and forwards kept frames byte for byte. `Policy.keep` is written ONCE against `CanGetOption` and runs on bytes (`Wire` givens) and on a decoded `Detail` (`Model` givens: eo-generics `prism`/`lens` chains, checked with `OptionalTests`); the differential tests compare the two on every message. Leaf prisms don't check the kind — the policy classifies first. `Wire.kind` is the one string path (`"type"` is the codec's tag, not a model field) and the one construction-time throw; `Issue.decode` the one catch of jsoniter-scala's throwing decode. `GrpcFrames.Malformed` fails the stream (compressed / oversized / truncated frames): no HTTP/2, no compression. Sockets are read 256 KiB at a time (`Net.reads`): fs2's 8 KiB `reads` was the loopback bottleneck. `Issues` = ScalaCheck generators DERIVED from the model by scalacheck-derived (only leaves hand-written: text, JSON-safe doubles, the opaque `Module` / `Votes`); `Issues(seed)` samples at `Issues.Size` (7, ~4 KB/message) with `Seed.slide` — consecutive `Seed.next` seeds draw overlapping, correlated values; derived sums shrink the size by one per level and fail at 0, so properties start at `minSize = 2`. Not in the root aggregate; CI runs `demoGrpcFilter/test` | +| `demoGrpcFilterBench` | `demos/grpc-filter-bench/` | — (not published) | JMH for that demo: one message judged (full decode vs optics, with `detail` late as written or moved first) and the in-memory filter pipe. Numbers live on `site/docs/cookbook/grpc-filter.md` (the cookbook's "Allocation-free stream triage demo" page). CI only compiles it (`demoGrpcFilterBench/Jmh/compile`) | The root project aggregates `core`, `laws`, `tests`, `generics`, `schemes`, `schemesLaws`, `circe`, `avro`, `jsoniter`, `zio`, and `kyo`. `sbt compile` and `sbt test` -cover those; benchmarks and the `demoUplcJvm` demo must be invoked explicitly -(see below; `sbt demoUplcJvm/test`). +cover those; benchmarks and the demos must be invoked explicitly +(see below; `sbt demoUplcJvm/test`, `sbt demoGrpcFilter/test`). Published modules compile with Scala 3's bytecode optimizer (`scala3LibrarySettings` in `build.sbt`): `-opt`, plus `-opt-inline:dev.constructive.eo.**` — the inliner may copy code @@ -76,6 +78,8 @@ sbt test # run all test suites (core, tests, generi sbt "~compile" # watch compile scalafmt # format in place per .scalafmt.conf scalafmt --check # CI-style verify + # (`project.git = true`: untracked files are skipped — + # `git add` new files before formatting them) scalafix --rules RemoveUnused # example rewrite scala -e 'println(1 + 2)' # quick REPL eval ``` diff --git a/README.md b/README.md index 47c3309e..f1d5af56 100644 --- a/README.md +++ b/README.md @@ -138,7 +138,7 @@ for cats typeclasses. - jsoniter-scala integration: - ZIO integration: - Kyo integration (Records, kyo-schema bridge): -- Cookbook (recipes): +- Cookbook (recipes): - Demo — a UPLC (Cardano Plutus Core) → JVM bytecode compiler built as one eo `Iso`, checked against the Scalus CEK machine: [`demos/uplc-jvm`](./demos/uplc-jvm/README.md) diff --git a/build.sbt b/build.sbt index 8c982fdc..d103453f 100644 --- a/build.sbt +++ b/build.sbt @@ -68,6 +68,8 @@ ThisBuild / githubWorkflowBuildPreamble ++= Seq( "demoUplcJvm/scalafmtCheckAll", "demoUplcJvmValidators/scalafmtCheckAll", "demoUplcJvmBench/scalafmtCheckAll", + "demoGrpcFilter/scalafmtCheckAll", + "demoGrpcFilterBench/scalafmtCheckAll", ), name = Some("Check formatting"), ), @@ -77,6 +79,8 @@ ThisBuild / githubWorkflowBuildPreamble ++= Seq( "demoUplcJvm/scalafixAll --check", "demoUplcJvmValidators/scalafixAll --check", "demoUplcJvmBench/scalafixAll --check", + "demoGrpcFilter/scalafixAll --check", + "demoGrpcFilterBench/scalafixAll --check", ), name = Some("Check scalafix"), ), @@ -89,6 +93,10 @@ ThisBuild / githubWorkflowBuild += WorkflowStep.Sbt( List("demoUplcJvm/test", "demoUplcJvmBench/Jmh/compile"), name = Some("Test UPLC → JVM demo (differential vs Scalus CEK)"), ) +ThisBuild / githubWorkflowBuild += WorkflowStep.Sbt( + List("demoGrpcFilter/test", "demoGrpcFilterBench/Jmh/compile"), + name = Some("Test gRPC filter demo (eo-jsoniter + fs2)"), +) // ------------------------------------------------------------------- // Scalafix wiring. SemanticDB exports are required by the semantic @@ -986,7 +994,7 @@ lazy val docs: Project = project mdocExtraArguments ++= Seq( "--scalac-options", // 120s: the 60s budget still tripped intermittently inside cold-JVM - // pre-commit hook sessions (cookbook.md avro fences on a loaded box). + // pre-commit hook sessions (the cookbook's avro fences on a loaded box). // The `s` suffix is required — kindlings' DurationPattern rejects a bare // integer and silently reverts to the 5s default (see the ThisBuild // scalacOptions comment above). @@ -1225,6 +1233,57 @@ lazy val demoUplcJvmBench: Project = project publish / skip := true, ) +// ------------------------------------------------------------------- +// Demo: triage a gRPC stream of JSON messages straight off a socket with +// eo-jsoniter optics and fs2 — every message is classified and filtered by +// reading only the few leaves the policy needs, never decoding the 64-field +// message, and kept messages are forwarded byte for byte. Unpublished and OUT +// of the root aggregate (CI runs it as its own step); run with +// `sbt demoGrpcFilter/test` or `sbt demoGrpcFilter/run`. +// ------------------------------------------------------------------- +lazy val Fs2Version = "3.14.0" +lazy val fs2Io = "co.fs2" %% "fs2-io" % Fs2Version +// Arbitrary / Cogen derivation from Scala 3 mirrors: the demo's issues are drawn from generators +// derived from the model. +lazy val scalacheckDerived = "io.github.martinhh" %% "scalacheck-derived" % "0.10.0" + +lazy val demoGrpcFilter: Project = project + .in(file("demos/grpc-filter")) + .dependsOn( + LocalProject("core"), + LocalProject("generics"), + LocalProject("jsoniterIntegration"), + // OptionalTests: the model's optics are checked against the library's own rule set. + LocalProject("laws") % Test, + ) + .settings(commonSettings *) + .settings( + name := "cats-eo-demo-grpc-filter", + publish / skip := true, + libraryDependencies += fs2Io, + // The message model's codecs: the source encodes whole messages, the baseline decodes them. + libraryDependencies += jsoniterMacros, + // The stream's issues come from ScalaCheck generators derived from the model, in main code: + // the demo's upstream service and the benchmarks draw from them too. + libraryDependencies += scalacheck, + libraryDependencies += scalacheckDerived, + libraryDependencies += discipline % Test, + Compile / run / fork := true, + ) + +// JMH for the gRPC filter demo: full decode against eo-jsoniter triage, per message and per +// stream. Results are reported on the demo's docs page, like the UPLC demo's. +// Run with `sbt "demoGrpcFilterBench/Jmh/run -prof gc"`. +lazy val demoGrpcFilterBench: Project = project + .in(file("demos/grpc-filter-bench")) + .enablePlugins(JmhPlugin) + .dependsOn(LocalProject("demoGrpcFilter")) + .settings(commonSettings *) + .settings( + name := "cats-eo-demo-grpc-filter-bench", + publish / skip := true, + ) + // Single source of truth for the JMH invocation (the config that the per-class // annotations pin — see JmhDefaults' Phase-4 note for why that preamble can't be // DRY'd). Append a filter: `sbt "bench .*OrderAvroBench.*"`. diff --git a/demos/grpc-filter-bench/src/main/scala/dev/constructive/eo/demo/grpc/bench/StreamBench.scala b/demos/grpc-filter-bench/src/main/scala/dev/constructive/eo/demo/grpc/bench/StreamBench.scala new file mode 100644 index 00000000..0547d08e --- /dev/null +++ b/demos/grpc-filter-bench/src/main/scala/dev/constructive/eo/demo/grpc/bench/StreamBench.scala @@ -0,0 +1,57 @@ +package dev.constructive.eo.demo.grpc +package bench + +import scala.compiletime.uninitialized + +import cats.effect.unsafe.implicits.global +import cats.effect.{IO, Ref} +import com.github.plokhotnyuk.jsoniter_scala.core.writeToArray +import fs2.{Chunk, Stream} +import java.util.concurrent.TimeUnit +import org.openjdk.jmh.annotations.* + +/** The whole filter, per message, in memory: gRPC frames arriving in 256 KiB pieces (as the demo + * reads its socket) are cut into messages, judged, tallied, and the kept ones framed again. No + * sockets, so what differs between the two `judge`s is what each costs in a running pipeline. + */ +@State(Scope.Benchmark) +@BenchmarkMode(Array(Mode.AverageTime)) +@OutputTimeUnit(TimeUnit.NANOSECONDS) +@Fork(3) +@Warmup(iterations = 3, time = 2) +@Measurement(iterations = 5, time = 2) +@OperationsPerInvocation(StreamBench.Messages) +class StreamBench: + + @Param(Array("optics", "decode")) + var judge: String = uninitialized + + private var verdict: Array[Byte] => Verdict = uninitialized + + private val pieces: Vector[Chunk[Byte]] = + val framed = Stream + .emits(Issues(seed = 7).take(StreamBench.Messages).map(writeToArray(_)).toSeq) + .through(GrpcFrames.joinFrames) + .compile + .to(Array) + framed.grouped(256 * 1024).map(Chunk.array(_)).toVector + + @Setup def prepare(): Unit = + verdict = + if judge == "optics" then Triage.optics(Policy.default) else Triage.decoding(Policy.default) + + @Benchmark def filter: Long = + Ref + .of[IO, Tally](Tally.empty) + .flatMap(tally => + Stream + .emits(pieces) + .flatMap(Stream.chunk) + .through(Triage.filter(verdict, tally)) + .compile + .count + ) + .unsafeRunSync() + +object StreamBench: + final val Messages = 1024 diff --git a/demos/grpc-filter-bench/src/main/scala/dev/constructive/eo/demo/grpc/bench/TriageBench.scala b/demos/grpc-filter-bench/src/main/scala/dev/constructive/eo/demo/grpc/bench/TriageBench.scala new file mode 100644 index 00000000..f944e4e9 --- /dev/null +++ b/demos/grpc-filter-bench/src/main/scala/dev/constructive/eo/demo/grpc/bench/TriageBench.scala @@ -0,0 +1,66 @@ +package dev.constructive.eo.demo.grpc +package bench + +import scala.compiletime.uninitialized + +import com.github.plokhotnyuk.jsoniter_scala.core.{readFromArray, writeToArray} +import dev.constructive.eo.optics.Optic.* +import java.nio.charset.StandardCharsets.UTF_8 +import java.util.concurrent.TimeUnit +import org.openjdk.jmh.annotations.* +import org.openjdk.jmh.infra.Blackhole + +/** One message triaged, two ways, over a pool of messages small enough to stay in cache (as a + * message just read from a socket is): + * + * - `decode` — decode the whole `Issue`, then run the policy on its `Detail`. + * - `optics` — slice out the detail with eo-jsoniter, then run the same policy on its bytes. + * - `detail` — only the slicing: the one scan through the message. + * + * `layout` is where `detail` sits: `late`, where the codec writes it (61st of 64 fields, about 90% + * of the way in), or `first`, moved to the front. Decoding reads every byte either way; the scan + * reads the bytes in front of the field. + */ +@State(Scope.Benchmark) +@BenchmarkMode(Array(Mode.AverageTime)) +@OutputTimeUnit(TimeUnit.NANOSECONDS) +@Fork(3) +@Warmup(iterations = 3, time = 1) +@Measurement(iterations = 5, time = 1) +@OperationsPerInvocation(TriageBench.Pool) +class TriageBench: + + @Param(Array("late", "first")) + var layout: String = uninitialized + + private val policy = Policy.default + private var messages: Array[Array[Byte]] = uninitialized + + @Setup def prepare(): Unit = + val issues = Issues(seed = 7).take(TriageBench.Pool).toArray + messages = issues.map { issue => + val message = writeToArray(issue) + val laidOut = if layout == "first" then detailFirst(message) else message + require(readFromArray[Issue](laidOut) == issue, "the layout changed the message") + laidOut + } + + /** The message with `detail` moved to the front: the same JSON object, its fields reordered. */ + private def detailFirst(message: Array[Byte]): Array[Byte] = + val json = new String(message, UTF_8) + val detail = s"\"detail\":${new String(Wire.detail.getOption(message).get.bytes, UTF_8)}" + s"{$detail,${json.drop(1).replace(s",$detail", "")}".getBytes(UTF_8) + + @Benchmark def decode(bh: Blackhole): Unit = + import Model.given + messages.foreach(m => bh.consume(policy.keep(readFromArray[Issue](m).detail))) + + @Benchmark def optics(bh: Blackhole): Unit = + import Wire.given + messages.foreach(m => bh.consume(Wire.detail.getOption(m).exists(policy.keep(_)))) + + @Benchmark def detail(bh: Blackhole): Unit = + messages.foreach(m => bh.consume(Wire.detail.getOption(m))) + +object TriageBench: + final val Pool = 256 diff --git a/demos/grpc-filter/README.md b/demos/grpc-filter/README.md new file mode 100644 index 00000000..140f03a0 --- /dev/null +++ b/demos/grpc-filter/README.md @@ -0,0 +1,103 @@ +# gRPC stream triage, with eo-jsoniter and fs2 + +A filter on a gRPC stream of JSON messages, read as raw bytes off a TCP socket. +It classifies each message and forwards only the ones a policy keeps, without +ever decoding a message: it reads the three or so values the policy needs +straight from the bytes, with eo-jsoniter's prisms, and forwards what it keeps +byte for byte. + +``` +upstream service ──TCP──▶ filter ──TCP──▶ downstream service + │ + bytes ─ GrpcFrames.splitFrames ─▶ message ─ Wire.detail ⇀ DetailJson ─ Policy.keep ─▶ keep? + │ + bytes ◀─ GrpcFrames.joinFrames ◀─ the kept messages, byte for byte ◀────────────────────┘ +``` + +Each message is an `Issue` with 64 fields. The 61st, `detail`, is one of three +kinds, each with its own fields, nested up to three objects deep: + +```scala +enum Detail: + case Bug(component: String, build: String, reproducible: Boolean, trace: Trace) + case Incident(severity: Severity, startedAt: Long, mitigated: Boolean, impact: Impact) + case Request(product: String, votes: Votes, quarter: String, requester: Customer) +``` + +The policy keeps severe incidents on critical services, bugs whose top stack +frame is in a watched module, and popular requests from customers on the plans +that matter. It is written once, against capabilities: + +```scala +def keep[D](detail: D)(using + kind: CanGetOption[D, Kind], + severity: CanGetOption[D, Severity], + tier: CanGetOption[D, Tier], + // … module, plan, votes +): Boolean +``` + +With `Wire`'s givens it reads a detail's bytes: + +```scala +val detail: Focus[Array[Byte], Wire.DetailJson] = JsoniterPrism[Issue].field(_.detail).raw +given tier: Focus[DetailJson, Tier] = + JsoniterPrism[Detail.Incident].field(_.impact).field(_.service).field(_.tier) +``` + +With `Model`'s, a decoded `Detail`, through eo-generics' prisms and lenses: + +```scala +given tier: Focus[Detail, Tier] = + prism[Detail, Detail.Incident] + .andThen(lens[Detail.Incident](_.impact)) + .andThen(lens[Impact](_.service)) + .andThen(lens[Service](_.tier)) +``` + +The baseline decodes every message and uses `Model`. The tests check that +both judge every message alike. + +| file | what it is | +|---|---| +| `Issue.scala` | the message model and its jsoniter-scala codecs | +| `Issues.scala` | ScalaCheck generators for issues, derived from the model by scalacheck-derived | +| `Wire.scala` | eo-jsoniter optics over a message's bytes | +| `Model.scala` | the same foci on a decoded `Detail` (eo-generics) | +| `Policy.scala` | which issues to keep, written against `CanGetOption` | +| `GrpcFrames.scala` | gRPC's Length-Prefixed-Message framing as fs2 pipes: `splitFrames`, `joinFrames` | +| `Triage.scala` | a message's verdict, both ways; the filter pipe; its tally | +| `Net.scala` | the upstream service, the filter and the downstream service on TCP sockets | +| `Demo.scala` | runs the three on loopback, with each filter | + +## Numbers + +JMH (`demos/grpc-filter-bench`) on a 4-vCPU 2.1 GHz Xeon container, JDK 21: + +| per message | full decode | eo-jsoniter optics | +|---|--:|--:| +| judging one message | 13.8 µs, 8 557 B | 5.9 µs, 705 B | +| … with `detail` first in the message | 13.4 µs, 8 548 B | 1.2 µs, 664 B | +| the whole filter pipe, in memory | 17.7 µs, 13 917 B | 8.9 µs, 6 001 B | + +Over loopback sockets, `sbt "demoGrpcFilter/run 200000"` filters 91 k +messages/s with the optics and 48 k with full decoding. + +The scan through a message costs most of the optics' time. It reads the bytes +in front of the focused field, so it depends on where the producer puts that +field. + +## Limits + +The frames travel on plain TCP, not inside HTTP/2 streams. So there are no +headers, trailers or status codes, and no compression: a compressed frame +fails the stream. See [the docs page](../../site/docs/cookbook/grpc-filter.md) for +the details. + +## Running it + +```sh +sbt demoGrpcFilter/test # the suite +sbt "demoGrpcFilter/run 200000" # the three services on loopback +sbt "demoGrpcFilterBench/Jmh/run -prof gc" # the benchmarks +``` diff --git a/demos/grpc-filter/src/main/scala/dev/constructive/eo/demo/grpc/Demo.scala b/demos/grpc-filter/src/main/scala/dev/constructive/eo/demo/grpc/Demo.scala new file mode 100644 index 00000000..dcdaf857 --- /dev/null +++ b/demos/grpc-filter/src/main/scala/dev/constructive/eo/demo/grpc/Demo.scala @@ -0,0 +1,59 @@ +package dev.constructive.eo.demo.grpc + +import scala.concurrent.duration.FiniteDuration + +import cats.effect.{ExitCode, IO, IOApp} +import cats.syntax.all.* +import com.github.plokhotnyuk.jsoniter_scala.core.writeToArray +import fs2.Stream + +/** Runs the three parties on the loopback interface: once with the filter reading messages through + * eo-jsoniter optics, once decoding every message, as a baseline. The upstream service repeats a + * pool of 1 024 encoded issues, so encoding isn't what's measured, and each filter runs once + * untimed first: the JIT compiler takes a few hundred thousand messages to finish its work. + * + * {{{ + * sbt "demoGrpcFilter/run 200000" + * }}} + */ +object Demo extends IOApp: + + def run(args: List[String]): IO[ExitCode] = + val count = args.headOption.flatMap(_.toIntOption).getOrElse(200_000) + val pool = Issues(seed = 2026).take(1024).map(writeToArray(_)).toVector + val policy = Policy.default + val filters = List( + "eo-jsoniter optics" -> Triage.optics(policy), + "full decode (baseline)" -> Triage.decoding(policy), + ) + + def once(count: Int, name: String, verdict: Array[Byte] => Verdict): IO[String] = + val messages = Stream.emits(pool).repeat.take(count.toLong).covary[IO] + (Net.source(messages), Net.sink[IO, Received](Received.empty)(_.add(_, policy))).tupled.use { + case (upstream, (downstream, received)) => + for + started <- IO.monotonic + tally <- Net.filter[IO](upstream, downstream, verdict) + took <- IO.monotonic.map(_ - started) + got <- received + yield report(name, tally, took, got) + } + + IO.println("Warming up: one untimed run of each filter…") >> + filters.traverse_((name, verdict) => once(count, name, verdict)) >> + filters + .traverse_((name, verdict) => once(count, name, verdict).flatMap(IO.println(_))) + .as( + ExitCode.Success + ) + + private def report(name: String, tally: Tally, took: FiniteDuration, got: Received): String = + val seconds = took.toNanos / 1e9 + val megabytes = tally.bytes / 1e6 + val judging = tally.judging.toNanos / 1e3 / tally.messages + s"""|$name: ${tally.messages} messages, ${f"$megabytes%.1f"} MB in ${f"$seconds%.2f"} s: + | ${f"${tally.messages / seconds / 1000}%.0f"} k messages/s, ${f"${megabytes / seconds}%.0f"} MB/s + | judging them took ${f"$judging%.1f"} µs per message; the rest is sockets and framing + |${tally.show} + | downstream received ${got.messages} messages (${tally.keptMessages} kept); ${got.issues} decode as issues, and the policy keeps ${got.kept} of them + |""".stripMargin diff --git a/demos/grpc-filter/src/main/scala/dev/constructive/eo/demo/grpc/GrpcFrames.scala b/demos/grpc-filter/src/main/scala/dev/constructive/eo/demo/grpc/GrpcFrames.scala new file mode 100644 index 00000000..12f10be1 --- /dev/null +++ b/demos/grpc-filter/src/main/scala/dev/constructive/eo/demo/grpc/GrpcFrames.scala @@ -0,0 +1,85 @@ +package dev.constructive.eo.demo.grpc + +import scala.util.control.NoStackTrace + +import cats.syntax.all.* +import fs2.{Chunk, Pipe, Pull, RaiseThrowable, Stream} + +/** gRPC's message framing, as fs2 pipes over raw bytes. The gRPC over HTTP/2 spec frames each + * message as a Length-Prefixed-Message: one byte saying whether the message is compressed, its + * length as a 4-byte big-endian unsigned integer, then the message. Over HTTP/2 a stream of such + * frames is the body of a gRPC call; in this demo they come straight off a TCP socket, in whatever + * pieces the socket reads them. + */ +object GrpcFrames: + + /** The compressed flag and the length. */ + val HeaderBytes = 5 + + /** gRPC's default limit on a received message: 4 MiB. */ + val DefaultMaxMessageBytes: Int = 4 * 1024 * 1024 + + /** The bytes are not gRPC frames this decoder can read. It fails the stream, like the socket + * errors around it: a gRPC peer answers a malformed frame by ending the call. + */ + final case class Malformed(reason: String) extends RuntimeException(reason) with NoStackTrace + + /** Raw bytes → one array per message: the message's bytes, uncompressed, without its header. + * Messages may straddle the incoming chunks any way at all. Fails with [[Malformed]] on a + * compressed message (no message encoding is negotiated here), on one longer than + * `maxMessageBytes`, or when the bytes end inside a frame. + */ + def splitFrames[F[_]: RaiseThrowable]( + maxMessageBytes: Int = DefaultMaxMessageBytes + ): Pipe[F, Byte, Array[Byte]] = + def go(buffered: Chunk[Byte], rest: Stream[F, Byte]): Pull[F, Array[Byte], Unit] = + split(buffered, maxMessageBytes) match + case Left(reason) => Pull.raiseError[F](Malformed(reason)) + case Right((messages, left)) => + Pull.output(messages) >> rest.pull.uncons.flatMap { + case Some((chunk, tail)) => go(left ++ chunk, tail) + case None if left.isEmpty => Pull.done + case None => + Pull.raiseError[F](Malformed(s"the bytes end inside a frame, ${left.size} bytes in")) + } + in => go(Chunk.empty, in).stream + + /** Messages → their frames, uncompressed. The messages' bytes are not copied. */ + def joinFrames[F[_]]: Pipe[F, Array[Byte], Byte] = + _.mapChunks(_.flatMap(frame)) + + /** One message's frame: not compressed, its length, the message. */ + def frame(message: Array[Byte]): Chunk[Byte] = + val length = message.length + Chunk(0, length >>> 24, length >>> 16, length >>> 8, length).map(_.toByte) ++ + Chunk.array(message) + + /** Every whole frame at the front of `bytes`, as messages, and the bytes after them. */ + private def split( + bytes: Chunk[Byte], + maxMessageBytes: Int, + ): Either[String, (Chunk[Array[Byte]], Chunk[Byte])] = + (List.empty[Array[Byte]], bytes).tailRecM { (messages, rest) => + unframe(rest, maxMessageBytes).map { + case Some((message, next)) => Left((message :: messages, next)) + case None => Right((Chunk.from(messages.reverse), rest)) + } + } + + /** The message in the frame at the front of `bytes`, and the bytes after it; `None` until the + * whole frame is there. + */ + private def unframe( + bytes: Chunk[Byte], + maxMessageBytes: Int, + ): Either[String, Option[(Array[Byte], Chunk[Byte])]] = + if bytes.size < HeaderBytes then Right(None) + else + val length = (1 to 4).foldLeft(0L)((n, i) => n << 8 | (bytes(i) & 0xffL)) + if bytes(0) != 0 then Left(s"compressed flag ${bytes(0)}: no message encoding negotiated") + else if length > maxMessageBytes then Left(s"a $length-byte message: over $maxMessageBytes") + else + Right(Option.when(bytes.size - HeaderBytes >= length) { + val (message, rest) = bytes.drop(HeaderBytes).splitAt(length.toInt) + (message.toArray, rest) + }) diff --git a/demos/grpc-filter/src/main/scala/dev/constructive/eo/demo/grpc/Issue.scala b/demos/grpc-filter/src/main/scala/dev/constructive/eo/demo/grpc/Issue.scala new file mode 100644 index 00000000..039f7c34 --- /dev/null +++ b/demos/grpc-filter/src/main/scala/dev/constructive/eo/demo/grpc/Issue.scala @@ -0,0 +1,197 @@ +package dev.constructive.eo.demo.grpc + +import scala.util.control.NonFatal + +import com.github.plokhotnyuk.jsoniter_scala.core.{ + readFromArray, + JsonReader, + JsonValueCodec, + JsonWriter +} +import com.github.plokhotnyuk.jsoniter_scala.macros.{CodecMakerConfig, JsonCodecMaker} + +/** One message of the stream: an issue, as an issue tracker publishes it. 64 fields. The 61st, + * `detail`, is one of three kinds of issue, each with its own fields, nested up to three objects + * deep (`detail` → `impact` → `service`). The demo never builds one of these from the wire: the + * source encodes them, and the tests and benchmarks decode them as the baseline the optics are + * checked and measured against. + */ +final case class Issue( + id: Long, + key: String, + project: String, + title: String, + description: String, + status: Status, + priority: Priority, + resolution: Option[String], + reporter: Person, + assignee: Option[Person], + watchers: List[String], + createdAt: Long, + updatedAt: Long, + resolvedAt: Option[Long], + dueAt: Option[Long], + estimateMinutes: Int, + spentMinutes: Int, + storyPoints: Double, + sprint: String, + epic: String, + fixVersion: String, + affectedVersions: List[String], + components: List[String], + labels: List[String], + rank: Long, + parentId: Option[Long], + subtaskIds: List[Long], + links: List[Link], + attachments: List[Attachment], + environment: Environment, + browser: String, + locale: String, + timezone: String, + region: String, + tenantId: String, + customerId: String, + source: String, + channel: String, + externalUrl: String, + commitSha: String, + branch: String, + buildNumber: Int, + pipelineId: Long, + views: Int, + commentCount: Int, + reopenCount: Int, + duplicateOf: Option[Long], + confidential: Boolean, + security: Boolean, + flagged: Boolean, + archived: Boolean, + slaPolicy: String, + slaBreached: Boolean, + cost: Double, + currency: String, + score: Double, + checksum: String, + schemaVersion: Int, + traceId: String, + spanId: String, + detail: Detail, + tags: Map[String, String], + checklist: List[CheckItem], + revision: Long, +) + +object Issue: + /** The whole message, every field: what the source writes and the baseline reads. */ + given JsonValueCodec[Issue] = JsonCodecMaker.make + + /** The whole message decoded, if it decodes. jsoniter-scala reports bytes that don't decode by + * throwing: this is the one place that becomes a value. + */ + def decode(message: Array[Byte]): Option[Issue] = + try Some(readFromArray[Issue](message)) + catch case NonFatal(_) => None + +/** The three kinds of issue. On the wire, `"type"` names the case: jsoniter-scala writes it as the + * object's first field. + */ +enum Detail: + case Bug(component: String, build: String, reproducible: Boolean, trace: Trace) + case Incident(severity: Severity, startedAt: Long, mitigated: Boolean, impact: Impact) + case Request(product: String, votes: Votes, quarter: String, requester: Customer) + +object Detail: + given JsonValueCodec[Detail] = JsonCodecMaker.make + +/** Which of the three kinds a message is: the classification. Spelled as `Detail`'s cases, so it + * decodes straight from the `"type"` field. + */ +enum Kind: + case Bug, Incident, Request + +object Kind: + given JsonValueCodec[Kind] = JsonCodecMaker.make(asString) + + def of(detail: Detail): Kind = detail match + case _: Detail.Bug => Bug + case _: Detail.Incident => Incident + case _: Detail.Request => Request + +enum Status: + case Open, Triaged, InProgress, Blocked, Resolved, Closed + +object Status: + given JsonValueCodec[Status] = JsonCodecMaker.make(asString) + +enum Priority: + case P0, P1, P2, P3, P4 + +object Priority: + given JsonValueCodec[Priority] = JsonCodecMaker.make(asString) + +enum Severity: + case Sev1, Sev2, Sev3, Sev4 + +object Severity: + given JsonValueCodec[Severity] = JsonCodecMaker.make(asString) + +/** How much depends on a service. */ +enum Tier: + case Critical, Standard, Internal + +object Tier: + given JsonValueCodec[Tier] = JsonCodecMaker.make(asString) + +/** A customer's plan. */ +enum Plan: + case Free, Team, Enterprise + +object Plan: + given JsonValueCodec[Plan] = JsonCodecMaker.make(asString) + +/** Codec settings for an enum without fields: its case's name, as a JSON string. */ +private inline def asString: CodecMakerConfig = CodecMakerConfig.withDiscriminatorFieldName(None) + +/** The code module a stack frame is in. */ +opaque type Module = String + +object Module: + def apply(name: String): Module = name + + extension (module: Module) def name: String = module + + given JsonValueCodec[Module] = new JsonValueCodec[Module]: + def decodeValue(in: JsonReader, default: Module): Module = in.readString(default) + def encodeValue(module: Module, out: JsonWriter): Unit = out.writeVal(module) + def nullValue: Module = null.asInstanceOf[Module] + +/** How many customers asked for a feature. */ +opaque type Votes = Int + +object Votes: + def apply(count: Int): Votes = count + + extension (votes: Votes) def count: Int = votes + + given JsonValueCodec[Votes] = new JsonValueCodec[Votes]: + def decodeValue(in: JsonReader, default: Votes): Votes = in.readInt() + def encodeValue(votes: Votes, out: JsonWriter): Unit = out.writeVal(votes) + def nullValue: Votes = 0 + +final case class Person(name: String, email: String, team: Team) +final case class Team(name: String, lead: String, org: Org) +final case class Org(name: String, region: String, costCenter: String) +final case class Link(kind: String, target: String) +final case class Attachment(name: String, bytes: Long, mime: String) +final case class Environment(os: String, arch: String, runtime: Runtime) +final case class Runtime(name: String, version: String) +final case class CheckItem(text: String, done: Boolean) + +final case class Trace(exception: String, message: String, frames: List[Frame]) +final case class Frame(module: Module, file: String, line: Int, function: String) +final case class Impact(users: Long, errorRate: Double, service: Service) +final case class Service(name: String, tier: Tier, region: String) +final case class Customer(name: String, seats: Int, contract: Contract) +final case class Contract(plan: Plan, renewsAt: Long, value: Double) diff --git a/demos/grpc-filter/src/main/scala/dev/constructive/eo/demo/grpc/Issues.scala b/demos/grpc-filter/src/main/scala/dev/constructive/eo/demo/grpc/Issues.scala new file mode 100644 index 00000000..cfb58048 --- /dev/null +++ b/demos/grpc-filter/src/main/scala/dev/constructive/eo/demo/grpc/Issues.scala @@ -0,0 +1,67 @@ +package dev.constructive.eo.demo.grpc + +import io.github.martinhh.derived.scalacheck.deriveArbitrary +import org.scalacheck.rng.Seed +import org.scalacheck.{Arbitrary, Gen} + +/** The stream's content: ScalaCheck generators for issues, derived from the model by + * scalacheck-derived, one per case class and enum, from their fields and cases. A third of the + * issues are of each kind. Only the leaves are written by hand: text, numbers JSON can carry, and + * the two opaque types. + */ +object Issues: + + private def words(spaced: String): Vector[String] = spaced.split(' ').toVector + + private val word = Gen.oneOf( + words( + "checkout latency timeout retry customer invoice cache deploy rollback queue partition " + + "replica token session webhook refund ledger balance report export import search index " + + "shard throttle quota alert page dashboard metric trace span error warning config flag " + + "release build" + ) + ) + + private val unusual = Gen.oneOf( + words( + "\"quoted\" back\\slash line\nbreak tab\there \u0001control " + + "café Grüße naïve 漢字 日本語 🚀 emoji👍" + ) + ) + + /** The modules a stack frame can be in. */ + val modules: Vector[Module] = + words("payments ledger checkout search auth notifications catalog inventory shipping analytics") + .map(Module(_)) + + /** Up to `size` words. About one in twenty is text JSON must escape (quotes, backslashes, line + * breaks) or non-ASCII, so whatever reads the bytes meets them. + */ + given Arbitrary[String] = Arbitrary( + Gen + .sized(n => Gen.choose(0, n)) + .flatMap(Gen.listOfN(_, Gen.frequency(19 -> word, 1 -> unusual))) + .map(_.mkString(" ")) + ) + + /** JSON has no NaN or infinities. */ + given Arbitrary[Double] = Arbitrary(Gen.choose(0.0, 1e6)) + + given Arbitrary[Module] = Arbitrary(Gen.oneOf(modules)) + given Arbitrary[Votes] = Arbitrary(Gen.choose(0, 200).map(Votes(_))) + given Arbitrary[Detail] = deriveArbitrary + given Arbitrary[Issue] = deriveArbitrary + + /** The size the stream's issues are drawn at: up to this many words in a text, and elements in a + * list. About 4 KB of JSON per issue. + */ + val Size = 7 + + /** Issues drawn from `seed`, at [[Size]]: the same seed gives the same issues. Each issue gets a + * seed of its own (`slide`): consecutive seeds (`next`) would draw overlapping values. + */ + def apply(seed: Long): Iterator[Issue] = + val parameters = Gen.Parameters.default.withSize(Size) + Iterator + .iterate(Seed(seed))(_.slide) + .map(Arbitrary.arbitrary[Issue].pureApply(parameters, _)) diff --git a/demos/grpc-filter/src/main/scala/dev/constructive/eo/demo/grpc/Model.scala b/demos/grpc-filter/src/main/scala/dev/constructive/eo/demo/grpc/Model.scala new file mode 100644 index 00000000..440e2482 --- /dev/null +++ b/demos/grpc-filter/src/main/scala/dev/constructive/eo/demo/grpc/Model.scala @@ -0,0 +1,44 @@ +package dev.constructive.eo.demo.grpc + +import dev.constructive.eo.generics.{lens, prism} +import dev.constructive.eo.optics.{AffineFold, Optional, PickFold} + +/** The same foci as [[Wire]]'s, on a decoded `Detail`: what the policy reads when the whole message + * has been decoded, as the baseline does. Built from eo-generics' derived prisms (one per case) + * and lenses (one per field). + */ +object Model: + + given kind: PickFold[Detail, Kind] = AffineFold(detail => Some(Kind.of(detail))) + + given severity: Focus[Detail, Severity] = + prism[Detail, Detail.Incident].andThen(lens[Detail.Incident](_.severity)) + + given tier: Focus[Detail, Tier] = + prism[Detail, Detail.Incident] + .andThen(lens[Detail.Incident](_.impact)) + .andThen(lens[Impact](_.service)) + .andThen(lens[Service](_.tier)) + + given module: Focus[Detail, Module] = + prism[Detail, Detail.Bug] + .andThen(lens[Detail.Bug](_.trace)) + .andThen(lens[Trace](_.frames)) + .andThen(head[Frame]) + .andThen(lens[Frame](_.module)) + + given plan: Focus[Detail, Plan] = + prism[Detail, Detail.Request] + .andThen(lens[Detail.Request](_.requester)) + .andThen(lens[Customer](_.contract)) + .andThen(lens[Contract](_.plan)) + + given votes: Focus[Detail, Votes] = + prism[Detail, Detail.Request].andThen(lens[Detail.Request](_.votes)) + + /** A list's first element, if it has one. */ + private def head[A]: Optional[List[A], List[A], A, A] = + Optional( + as => as.headOption.toRight(as), + (as, a) => a :: as.drop(1), + ) diff --git a/demos/grpc-filter/src/main/scala/dev/constructive/eo/demo/grpc/Net.scala b/demos/grpc-filter/src/main/scala/dev/constructive/eo/demo/grpc/Net.scala new file mode 100644 index 00000000..a5ab0f42 --- /dev/null +++ b/demos/grpc-filter/src/main/scala/dev/constructive/eo/demo/grpc/Net.scala @@ -0,0 +1,101 @@ +package dev.constructive.eo.demo.grpc + +import cats.effect.syntax.all.* +import cats.effect.{Async, Ref, Resource} +import cats.syntax.all.* +import com.comcast.ip4s.* +import fs2.Stream +import fs2.io.net.{Network, Socket} + +/** What the downstream service received: messages and bytes, how many decoded as issues, and how + * many of those the policy keeps when it reads the decoded issue. + */ +final case class Received(messages: Long, bytes: Long, issues: Long, kept: Long): + + def add(message: Array[Byte], policy: Policy): Received = + import Model.given + val detail = Issue.decode(message).map(_.detail) + Received( + messages + 1, + bytes + message.length, + issues + detail.size, + kept + detail.count(policy.keep(_)), + ) + +object Received: + val empty: Received = Received(0, 0, 0, 0) + +/** The demo's three parties, each on its own TCP socket on the loopback interface: an upstream + * service streaming issues, the filter, and a downstream service that receives what the filter + * keeps. + */ +object Net: + + private val loopback = SocketAddress(ip"127.0.0.1", port"0") + + /** A socket's bytes, up to 256 KiB per read. fs2's `reads` asks for 8 KiB at a time, and every + * read is a trip through the event loop: on loopback that, not the filter, was the bottleneck. + */ + def reads[F[_]](socket: Socket[F]): Stream[F, Byte] = + Stream.repeatEval(socket.read(256 * 1024)).unNoneTerminate.unchunks + + /** The upstream service: to each client that connects, it streams `messages` as gRPC frames and + * then closes its side. Gives the address it listens on. + */ + def source[F[_]: {Network, Async}]( + messages: Stream[F, Array[Byte]] + ): Resource[F, GenSocketAddress] = + Network[F].bind(loopback).flatMap { server => + server + .accept + .map(client => + messages.through(GrpcFrames.joinFrames).through(client.writes) ++ + Stream.exec(client.endOfOutput) + ) + .parJoinUnbounded + .compile + .drain + .background + .as(server.address) + } + + /** The downstream service: accepts one client and folds every message it sends into `z` with `f`. + * Gives the address it listens on, and the result once the client is done. + */ + def sink[F[_]: {Network, Async}, A](z: A)( + f: (A, Array[Byte]) => A + ): Resource[F, (GenSocketAddress, F[A])] = + Network[F].bind(loopback).flatMap { server => + server + .accept + .head + .flatMap(reads(_).through(GrpcFrames.splitFrames())) + .fold(z)(f) + .compile + .lastOrError + .background + .map(outcome => (server.address, outcome.flatMap(_.embedError))) + } + + /** The filter: connects to both, triages the upstream's stream with `verdict` and writes the + * frames it keeps downstream. Gives the tally once the upstream's stream ends. Reading, judging + * and writing each run on their own fiber, with a few chunks buffered between them, so the + * sockets wait on the judging only when it is the slowest of the three. + */ + def filter[F[_]: {Network, Async}]( + upstream: GenSocketAddress, + downstream: GenSocketAddress, + verdict: Array[Byte] => Verdict, + ): F[Tally] = + (Network[F].connect(upstream), Network[F].connect(downstream)).tupled.use { (in, out) => + Ref.of[F, Tally](Tally.empty).flatMap { tally => + reads(in) + .prefetchN(4) + .through(Triage.filter(verdict, tally)) + .prefetchN(4) + .through(out.writes) + .compile + .drain >> + out.endOfOutput >> tally.get + } + } diff --git a/demos/grpc-filter/src/main/scala/dev/constructive/eo/demo/grpc/Policy.scala b/demos/grpc-filter/src/main/scala/dev/constructive/eo/demo/grpc/Policy.scala new file mode 100644 index 00000000..a2d3037b --- /dev/null +++ b/demos/grpc-filter/src/main/scala/dev/constructive/eo/demo/grpc/Policy.scala @@ -0,0 +1,47 @@ +package dev.constructive.eo.demo.grpc + +import dev.constructive.eo.CanGetOption + +/** Which issues to keep: urgent incidents on the services that matter, bugs whose top stack frame + * is in a watched module, and popular requests from customers on the plans that matter. Anything + * else, including a kind this policy doesn't know, is dropped. + * + * [[keep]] is written once, against capabilities: it asks for each leaf it reads and never says + * where the leaf comes from. The same code runs on a message's bytes (with [[Wire]]'s optics) and + * on a decoded `Detail` (with [[Model]]'s), which is how the tests check one against the other. + */ +final case class Policy( + urgent: Set[Severity], + tiers: Set[Tier], + modules: Set[Module], + plans: Set[Plan], + minVotes: Votes, +): + + def keep[D](detail: D)(using + kind: CanGetOption[D, Kind], + severity: CanGetOption[D, Severity], + tier: CanGetOption[D, Tier], + module: CanGetOption[D, Module], + plan: CanGetOption[D, Plan], + votes: CanGetOption[D, Votes], + ): Boolean = + kind.getOption(detail) match + case Some(Kind.Incident) => + severity.getOption(detail).exists(urgent) && tier.getOption(detail).exists(tiers) + case Some(Kind.Bug) => module.getOption(detail).exists(modules) + case Some(Kind.Request) => + plan.getOption(detail).exists(plans) && votes + .getOption(detail) + .exists(_.count >= minVotes.count) + case None => false + +object Policy: + + val default: Policy = Policy( + urgent = Set(Severity.Sev1, Severity.Sev2), + tiers = Set(Tier.Critical), + modules = Set(Module("payments"), Module("ledger")), + plans = Set(Plan.Enterprise), + minVotes = Votes(100), + ) diff --git a/demos/grpc-filter/src/main/scala/dev/constructive/eo/demo/grpc/Triage.scala b/demos/grpc-filter/src/main/scala/dev/constructive/eo/demo/grpc/Triage.scala new file mode 100644 index 00000000..282ccd21 --- /dev/null +++ b/demos/grpc-filter/src/main/scala/dev/constructive/eo/demo/grpc/Triage.scala @@ -0,0 +1,96 @@ +package dev.constructive.eo.demo.grpc + +import scala.concurrent.duration.{Duration, FiniteDuration} + +import cats.effect.syntax.all.* +import cats.effect.{Async, Ref} +import cats.syntax.all.* +import dev.constructive.eo.optics.Optic.* +import fs2.{Chunk, Pipe} + +/** What triage made of one message: its kind, if it has one this demo knows, and whether the policy + * keeps it. + */ +final case class Verdict(kind: Option[Kind], keep: Boolean) + +/** How a stream was triaged: messages and bytes seen; per kind, how many were seen and kept; and + * the time spent judging them, apart from reading and writing sockets. `unclassified` counts the + * messages without a kind: no `detail`, or one of a kind this demo doesn't know (from a newer + * producer, say). The policy drops those. + */ +final case class Tally( + messages: Long, + bytes: Long, + seen: Map[Kind, Long], + kept: Map[Kind, Long], + unclassified: Long, + judging: FiniteDuration, +): + + def keptMessages: Long = kept.values.sum + + /** A chunk of messages, their verdicts, and how long they took. */ + def add(judged: Chunk[(Array[Byte], Verdict)], took: FiniteDuration): Tally = + judged.foldLeft(copy(judging = judging + took)) { + case (tally, (message, verdict)) => + def bump(counts: Map[Kind, Long], kind: Kind) = counts.updated(kind, counts(kind) + 1) + tally.copy( + messages = tally.messages + 1, + bytes = tally.bytes + message.length, + seen = verdict.kind.fold(tally.seen)(bump(tally.seen, _)), + kept = verdict.kind.filter(_ => verdict.keep).fold(tally.kept)(bump(tally.kept, _)), + unclassified = tally.unclassified + verdict.kind.fold(1L)(_ => 0L), + ) + } + + def show: String = + val rows = Kind.values.toList.map(k => f" ${k.toString}%-10s${seen(k)}%10d${kept(k)}%10d") + (f" ${"kind"}%-10s${"seen"}%10s${"kept"}%10s" :: rows ::: + List(f" ${"none"}%-10s$unclassified%10d${0}%10d")).mkString("\n") + +object Tally: + + val empty: Tally = + val none = Kind.values.map(_ -> 0L).toMap + Tally(0, 0, none, none, 0, Duration.Zero) + +/** Classifying and filtering a stream of gRPC-framed issues. */ +object Triage: + + /** A message's verdict, read through [[Wire]]'s optics: one scan of the message for its `detail`, + * then the few leaves the policy wants from the detail's bytes. Nothing else is decoded. + */ + def optics(policy: Policy)(message: Array[Byte]): Verdict = + import Wire.given + Wire.detail.getOption(message) match + case Some(detail) => Verdict(Wire.kind.getOption(detail), policy.keep(detail)) + case None => Verdict(None, keep = false) + + /** The same verdict the usual way, as a baseline: decode the whole `Issue`, then ask the policy + * about its `Detail` through [[Model]]'s optics. A message that doesn't decode (a kind this + * model doesn't know fails the whole message) has no kind and is dropped. + */ + def decoding(policy: Policy)(message: Array[Byte]): Verdict = + import Model.given + Issue.decode(message).map(_.detail) match + case Some(detail) => Verdict(Some(Kind.of(detail)), policy.keep(detail)) + case None => Verdict(None, keep = false) + + /** The filter: raw bytes of gRPC frames in; the frames of the messages `verdict` keeps out, each + * message forwarded byte for byte. Every chunk of verdicts, and how long it took, goes into + * `tally`. + */ + def filter[F[_]: Async]( + verdict: Array[Byte] => Verdict, + tally: Ref[F, Tally] + ): Pipe[F, Byte, Byte] = + _.through(GrpcFrames.splitFrames()) + .chunks + .evalMap { messages => + Async[F].delay(messages.map(message => (message, verdict(message)))).timed.flatMap { + (took, judged) => + tally.update(_.add(judged, took)).as(judged.collect { case (m, Verdict(_, true)) => m }) + } + } + .unchunks + .through(GrpcFrames.joinFrames) diff --git a/demos/grpc-filter/src/main/scala/dev/constructive/eo/demo/grpc/Wire.scala b/demos/grpc-filter/src/main/scala/dev/constructive/eo/demo/grpc/Wire.scala new file mode 100644 index 00000000..fbf07bb9 --- /dev/null +++ b/demos/grpc-filter/src/main/scala/dev/constructive/eo/demo/grpc/Wire.scala @@ -0,0 +1,67 @@ +package dev.constructive.eo.demo.grpc + +import com.github.plokhotnyuk.jsoniter_scala.core.JsonValueCodec +import com.github.plokhotnyuk.jsoniter_scala.macros.JsonCodecMaker +import dev.constructive.eo.data.Affine +import dev.constructive.eo.jsoniter.JsoniterPrism +import dev.constructive.eo.optics.Optic + +/** A read-write optic whose focus may be missing: eo-jsoniter's prisms, and [[Model]]'s chains of a + * prism and lenses. + */ +type Focus[S, A] = Optic[S, S, A, A, Affine] + +/** eo-jsoniter optics over a message as it arrives: JSON bytes. Each one scans the bytes for its + * path and decodes only what it focuses, so no `Issue` is ever built. + * + * Classifying and filtering takes two steps. [[detail]] finds the message's `detail` (one scan, + * past the 60 fields in front of it) and hands out that object's own bytes. The givens then read + * the few leaves the policy needs from those few hundred bytes. + * + * The paths are typed: `.field(_.impact)` is checked against the model when this file compiles, so + * renaming a field breaks the build rather than the filter. + */ +object Wire: + + /** A message's `detail` object: its own JSON, sliced out of the message. A separate type, so the + * detail's optics can't be pointed at a whole message. + */ + opaque type DetailJson = Array[Byte] + + // Typing a path needs a codec for every object on it; only the leaf's ever decodes anything. + private given JsonValueCodec[Detail.Bug] = JsonCodecMaker.make + private given JsonValueCodec[Detail.Incident] = JsonCodecMaker.make + private given JsonValueCodec[Detail.Request] = JsonCodecMaker.make + private given JsonValueCodec[Trace] = JsonCodecMaker.make + private given JsonValueCodec[List[Frame]] = JsonCodecMaker.make + private given JsonValueCodec[Frame] = JsonCodecMaker.make + private given JsonValueCodec[Impact] = JsonCodecMaker.make + private given JsonValueCodec[Service] = JsonCodecMaker.make + private given JsonValueCodec[Customer] = JsonCodecMaker.make + private given JsonValueCodec[Contract] = JsonCodecMaker.make + + /** Message ⇀ its detail's bytes, copied verbatim; nothing decoded. */ + val detail: Focus[Array[Byte], DetailJson] = JsoniterPrism[Issue].field(_.detail).raw + + /** Which kind of issue: the `"type"` field that names `Detail`'s case. */ + given kind: Focus[DetailJson, Kind] = + // The one string path: `"type"` is the codec's tag, not a field of the model. A `Left` would be + // a typo in this line, and every test reads this optic. + JsoniterPrism.fromPath[Kind]("$.type").fold(e => throw IllegalArgumentException(e), identity) + + // Each case's leaves: the path from the case's own root, read off its own fields. + given severity: Focus[DetailJson, Severity] = JsoniterPrism[Detail.Incident].field(_.severity) + + given tier: Focus[DetailJson, Tier] = + JsoniterPrism[Detail.Incident].field(_.impact).field(_.service).field(_.tier) + + given module: Focus[DetailJson, Module] = + JsoniterPrism[Detail.Bug].field(_.trace).field(_.frames).at(0).field(_.module) + + given plan: Focus[DetailJson, Plan] = + JsoniterPrism[Detail.Request].field(_.requester).field(_.contract).field(_.plan) + + given votes: Focus[DetailJson, Votes] = JsoniterPrism[Detail.Request].field(_.votes) + + /** A copy of the bytes. */ + extension (detail: DetailJson) def bytes: Array[Byte] = detail.clone diff --git a/demos/grpc-filter/src/test/scala/dev/constructive/eo/demo/grpc/GrpcFramesSpec.scala b/demos/grpc-filter/src/test/scala/dev/constructive/eo/demo/grpc/GrpcFramesSpec.scala new file mode 100644 index 00000000..16f9781e --- /dev/null +++ b/demos/grpc-filter/src/test/scala/dev/constructive/eo/demo/grpc/GrpcFramesSpec.scala @@ -0,0 +1,80 @@ +package dev.constructive.eo.demo.grpc + +import scala.annotation.tailrec +import scala.language.implicitConversions + +import fs2.{Chunk, Fallible, Stream} +import org.scalacheck.{Arbitrary, Gen, Prop} +import org.specs2.ScalaCheck +import org.specs2.mutable.Specification + +/** gRPC framing over bytes that arrive in arbitrary pieces. */ +class GrpcFramesSpec extends Specification with ScalaCheck: + + import GrpcFramesSpec.pieces + + def decode( + chunks: List[Chunk[Byte]], + max: Int = GrpcFrames.DefaultMaxMessageBytes, + ): Either[Throwable, List[List[Byte]]] = + Stream + .emits(chunks) + .flatMap(Stream.chunk) + .covary[Fallible] + .through(GrpcFrames.splitFrames(max)) + .compile + .toList + .map(_.map(_.toList)) + + def framed(messages: List[Array[Byte]]): Array[Byte] = + Stream.emits(messages).through(GrpcFrames.joinFrames).compile.to(Array) + + def malformed(bytes: Int*)(reason: String) = + decode(List(Chunk.array(bytes.map(_.toByte).toArray)), max = 10) must beLeft( + GrpcFrames.Malformed(reason) + ) + + private val messages = Gen.listOf(Gen.containerOf[Array, Byte](Arbitrary.arbitrary[Byte])) + private val sizes = Gen.nonEmptyListOf(Gen.oneOf(Gen.choose(1, 8), Gen.choose(1, 300))) + + "messages come back whole however the bytes are cut" >> Prop.forAll(messages, sizes) { + (messages, sizes) => + decode(pieces(framed(messages), sizes)) must beRight(messages.map(_.toList)) + } + + "a frame is a zero, the length big-endian, then the message" >> { + GrpcFrames.frame(Array[Byte](7, 8, 9)).toList === List[Byte](0, 0, 0, 0, 3, 7, 8, 9) + } + + "no bytes, no messages; an empty message is a message" >> { + decode(Nil) must beRight(Nil) + decode(List(GrpcFrames.frame(Array.emptyByteArray))) must beRight(List(Nil)) + } + + "a message at the size limit passes; one byte more fails the stream" >> { + decode(List(GrpcFrames.frame(new Array[Byte](10))), max = 10) must beRight + malformed(0, 0, 0, 0, 11)("a 11-byte message: over 10") + } + + "the length is unsigned" >> malformed(0, 255, 255, 255, 255)("a 4294967295-byte message: over 10") + + "compressed messages fail the stream: no message encoding is negotiated" >> { + malformed(1, 0, 0, 0, 1, 42)("compressed flag 1: no message encoding negotiated") + malformed(2, 0, 0, 0, 1, 42)("compressed flag 2: no message encoding negotiated") + } + + "bytes that end inside a frame fail the stream" >> { + malformed(0, 0, 0)("the bytes end inside a frame, 3 bytes in") + malformed(0, 0, 0, 0, 4, 1, 2)("the bytes end inside a frame, 7 bytes in") + } + +object GrpcFramesSpec: + + /** `bytes` in pieces of `sizes` (cycled), as a socket might deliver them. */ + def pieces(bytes: Array[Byte], sizes: List[Int]): List[Chunk[Byte]] = + @tailrec def go(at: Int, next: LazyList[Int], acc: List[Chunk[Byte]]): List[Chunk[Byte]] = + if at >= bytes.length then acc.reverse + else + val size = next.head.min(bytes.length - at) + go(at + size, next.tail, Chunk.array(bytes, at, size) :: acc) + go(0, LazyList.continually(sizes).flatten, Nil) diff --git a/demos/grpc-filter/src/test/scala/dev/constructive/eo/demo/grpc/PipelineSpec.scala b/demos/grpc-filter/src/test/scala/dev/constructive/eo/demo/grpc/PipelineSpec.scala new file mode 100644 index 00000000..ed115efe --- /dev/null +++ b/demos/grpc-filter/src/test/scala/dev/constructive/eo/demo/grpc/PipelineSpec.scala @@ -0,0 +1,88 @@ +package dev.constructive.eo.demo.grpc + +import scala.concurrent.duration.* +import scala.language.implicitConversions + +import cats.effect.unsafe.implicits.global +import cats.effect.{IO, Ref} +import cats.syntax.all.* +import com.github.plokhotnyuk.jsoniter_scala.core.writeToArray +import fs2.{Chunk, Stream} +import org.scalacheck.{Gen, Prop} +import org.specs2.ScalaCheck +import org.specs2.mutable.Specification + +/** The filter end to end: in memory, cut into arbitrary pieces, and over real sockets. */ +class PipelineSpec extends Specification with ScalaCheck: + + private val policy = Policy.default + private val issues = Issues(seed = 5).take(400).toVector + private val messages = issues.map(writeToArray(_)) + + /** A Request whose kind no one here knows: the filter reads on past it, and drops it. */ + private val task = + val request = messages(issues.indexWhere(_.detail.isInstanceOf[Detail.Request])) + new String(request, "UTF-8") + .replace("\"type\":\"Request\"", "\"type\":\"Task\"") + .getBytes("UTF-8") + + private val upstream = messages :+ task + + /** What the filter must forward: the messages the decoded issues say to keep, as they are. */ + private val expected = upstream.filter(Triage.decoding(policy)(_).keep).map(_.toList) + + private def run[A](io: IO[A]): A = io.timeout(60.seconds).unsafeRunSync() + + private def checkTally(tally: Tally) = + (tally.messages === upstream.size.toLong) + .and(tally.bytes === upstream.map(_.length.toLong).sum) + .and(tally.keptMessages === expected.size.toLong) + .and(tally.unclassified === 1L) + .and(tally.seen.values.sum === issues.size.toLong) + + "in memory, however the bytes are cut, the filter forwards exactly the kept messages" >> + Prop + .forAll(Gen.nonEmptyListOf(Gen.choose(1, 20_000))) { sizes => + val bytes = Stream.emits(upstream).through(GrpcFrames.joinFrames).compile.to(Array) + val (tally, out) = run(Ref.of[IO, Tally](Tally.empty).flatMap { tally => + Stream + .emits(GrpcFramesSpec.pieces(bytes, sizes)) + .flatMap(Stream.chunk) + .covary[IO] + .through(Triage.filter(Triage.optics(policy), tally)) + .through(GrpcFrames.splitFrames()) + .map(_.toList) + .compile + .toVector + .product(tally.get) + .map(_.swap) + }) + (out === expected).and(checkTally(tally)) + } + .set(minTestsOk = 20) + + "over sockets: upstream service → filter → downstream service" >> { + val (tally, received) = run( + ( + Net.source[IO](Stream.emits(upstream)), + Net.sink[IO, Vector[List[Byte]]](Vector.empty)(_ :+ _.toList), + ).tupled.use { + case (source, (sink, received)) => + Net.filter[IO](source, sink, Triage.optics(policy)).product(received) + } + ) + (received === expected).and(checkTally(tally)) + } + + "a malformed frame fails the filter's stream" >> { + val compressed = Chunk.array(Array[Byte](1, 0, 0, 0, 1, 42)) + run(Ref.of[IO, Tally](Tally.empty).flatMap { tally => + Stream + .chunk(compressed) + .covary[IO] + .through(Triage.filter(Triage.optics(policy), tally)) + .compile + .drain + .attempt + }) must beLeft(GrpcFrames.Malformed("compressed flag 1: no message encoding negotiated")) + } diff --git a/demos/grpc-filter/src/test/scala/dev/constructive/eo/demo/grpc/TriageSpec.scala b/demos/grpc-filter/src/test/scala/dev/constructive/eo/demo/grpc/TriageSpec.scala new file mode 100644 index 00000000..1d1e7361 --- /dev/null +++ b/demos/grpc-filter/src/test/scala/dev/constructive/eo/demo/grpc/TriageSpec.scala @@ -0,0 +1,105 @@ +package dev.constructive.eo.demo.grpc + +import scala.language.implicitConversions + +import com.github.plokhotnyuk.jsoniter_scala.core.{readFromArray, writeToArray, WriterConfig} +import dev.constructive.eo.laws.OptionalLaws +import dev.constructive.eo.laws.discipline.OptionalTests +import dev.constructive.eo.optics.Optic.* +import io.github.martinhh.derived.scalacheck.{deriveArbitrary, deriveCogen} +import org.scalacheck.{Arbitrary, Cogen, Prop} +import org.specs2.ScalaCheck +import org.specs2.mutable.Specification +import org.typelevel.discipline.specs2.mutable.Discipline + +/** Differential: reading a message through the optics must judge it exactly as decoding it does. */ +class TriageSpec extends Specification with ScalaCheck with Discipline: + + import Issues.given + + private val issues = Issues(seed = 11).take(3000).toVector + private val messages = issues.map(writeToArray(_)) + + private def agree(policy: Policy)(message: Array[Byte]) = + Triage.optics(policy)(message) === Triage.decoding(policy)(message) + + // Derived like the issues: any sets of severities, tiers, modules and plans, any vote threshold. + given Arbitrary[Policy] = deriveArbitrary + + "an issue has 64 fields" >> (issues.head.productArity === 64) + + "the optics judge every message as decoding it does" >> { + "under the default policy, on 3 000 issues the size the stream's are" >> + forall(messages)(agree(Policy.default)) + "under any policy, on 300 of them" >> + Prop.forAll((policy: Policy) => forall(messages.take(300))(agree(policy))) + // From size 2 up: below that the derived enums have no case left to draw. + "under any policy, on any issue of any size" >> + Prop + .forAll((policy: Policy, issue: Issue) => agree(policy)(writeToArray(issue))) + .set(minTestsOk = 500, minSize = 2, maxSize = 40) + } + + "every kind is both kept and dropped, so every branch of the policy is exercised" >> { + val verdicts = messages.map(Triage.optics(Policy.default)) + forall(Kind.values.toSeq) { kind => + verdicts.filter(_.kind.contains(kind)).map(_.keep).distinct must contain(exactly(true, false)) + } + } + + "the detail the optics slice out is the detail's own JSON" >> + forall(issues.zip(messages)) { (issue, message) => + Wire.detail.getOption(message).map(d => readFromArray[Detail](d.bytes)) must beSome( + issue.detail + ) + } + + "whitespace between the tokens changes nothing" >> { + val pretty = issues.take(200).map(writeToArray(_, WriterConfig.withIndentionStep(2))) + forall(pretty)(agree(Policy.default)) + } + + "messages the policy can't judge are dropped, unclassified" >> { + val request = messages(issues.indexWhere(_.detail.isInstanceOf[Detail.Request])) + def edited(from: String, to: String) = + new String(request, "UTF-8").replace(from, to).getBytes("UTF-8") + val unclassified = Verdict(None, keep = false) + "a kind from a newer producer: the optics read on, a decoder rejects the message" >> { + val task = edited("\"type\":\"Request\"", "\"type\":\"Task\"") + Triage.optics(Policy.default)(task) === unclassified + Triage.decoding(Policy.default)(task) === unclassified + } + "no detail" >> { + Triage.optics(Policy.default)(edited("\"detail\":", "\"derail\":")) === unclassified + } + "not JSON" >> { + val junk = "{\"id\":".getBytes("UTF-8") + Triage.optics(Policy.default)(junk) === unclassified + Triage.decoding(Policy.default)(junk) === unclassified + } + } + + // The model's optics, which the baseline and these tests read through, are lawful. + given Arbitrary[Severity] = deriveArbitrary + given Arbitrary[Tier] = deriveArbitrary + given Arbitrary[Plan] = deriveArbitrary + given Cogen[Severity] = deriveCogen + given Cogen[Tier] = deriveCogen + given Cogen[Plan] = deriveCogen + given Cogen[Module] = Cogen[String].contramap(_.name) + given Cogen[Votes] = Cogen[Int].contramap(_.count) + + private def optional[A: Arbitrary: Cogen](name: String, focus: Focus[Detail, A]) = + checkAll( + s"Model.$name", + new OptionalTests[Detail, A]: + val laws = new OptionalLaws[Detail, A]: + val optional = focus + .optional, + ) + + optional("severity", Model.severity) + optional("tier", Model.tier) + optional("module", Model.module) + optional("plan", Model.plan) + optional("votes", Model.votes) diff --git a/demos/uplc-jvm/README.md b/demos/uplc-jvm/README.md index bee81258..752c2e41 100644 --- a/demos/uplc-jvm/README.md +++ b/demos/uplc-jvm/README.md @@ -197,7 +197,7 @@ class file there for `javap`. library's nightly sweep): the CEK machine against the generated class for every validator, the Fibonacci puzzle at growing `n`, and what translation itself costs. The numbers and how they were taken are on the -[docs page](../../site/docs/demos/uplc-jvm.md#benchmarks). +[docs page](../../site/docs/cookbook/uplc-jvm.md#benchmarks). ```sh sbt "demoUplcJvmBench/Jmh/run -prof gc" diff --git a/demos/uplc-jvm/src/test/scala/dev/constructive/eo/demo/uplc/Demo.scala b/demos/uplc-jvm/src/test/scala/dev/constructive/eo/demo/uplc/Demo.scala index da7b5b51..a0571d82 100644 --- a/demos/uplc-jvm/src/test/scala/dev/constructive/eo/demo/uplc/Demo.scala +++ b/demos/uplc-jvm/src/test/scala/dev/constructive/eo/demo/uplc/Demo.scala @@ -8,7 +8,7 @@ import org.objectweb.asm.util.{Textifier, TraceMethodVisitor} import scalus.uplc.eval.Result import scalus.uplc.{DeBruijn, DeBruijnedProgram} -/** The round trip, stage by stage — what the docs page (`site/docs/demos/uplc-jvm.md`) shows. +/** The round trip, stage by stage — what the docs page (`site/docs/cookbook/uplc-jvm.md`) shows. * * {{{sbt "demoUplcJvm/Test/runMain dev.constructive.eo.demo.uplc.Demo [dir-for-the-class-file]"}}} * diff --git a/site/docs/benchmarks.md b/site/docs/benchmarks.md index 00df0245..b8db37ba 100644 --- a/site/docs/benchmarks.md +++ b/site/docs/benchmarks.md @@ -34,7 +34,7 @@ table, and for the ratios everywhere, the signal holds. > sub-nanosecond truth. See [Reproducing](#reproducing) and > [`benchmarks/README.md`](https://github.com/Constructive-Programming/eo/blob/main/benchmarks/README.md). -The [UPLC → JVM demo](demos/uplc-jvm.md) has its own JMH suite — the Scalus CEK +The [blockchain smart contract compilation demo](cookbook/uplc-jvm.md) has its own JMH suite — the Scalus CEK machine against the class files the demo compiles — reported on its page, not here. diff --git a/site/docs/cookbook.md b/site/docs/cookbook.md deleted file mode 100644 index dee95edf..00000000 --- a/site/docs/cookbook.md +++ /dev/null @@ -1,1100 +0,0 @@ -# Cookbook - -Optics are probably the most powerful unused tool in our arsenal — -this page is the working proof. Runnable patterns for the questions -that come up most often, organised by the three jobs optics do best: - -1. **[Navigating structures](#navigating-structures)** — reach one - branch of a sum, every node of a recursive tree, several fields - at once, or a field inside bytes you never decode. One `.andThen` - surface for all of it. -2. **[Decoupling and modularity](#decoupling-and-modularity)** — - replace a direct dependence on a data type with an optic: depend - on the *relationship*, ask for the *weakest capability* you need, - and let independent modules interoperate at the call site. -3. **[Effect threading](#effect-threading)** — turn control flow - into type definitions: the optic pins *where*, the effect type - decides *what happens* — abort, accumulate, audit, defer. - -The framing follows Hansen — *We Need More Optics* -() and -Penner — *Optics By Example* -(). These are worked -examples, not a tutorial — for the optic families themselves see the -[Optics reference](optics.md). Every fence is compiled by mdoc -against the current library version, and every recipe cites its -source. - -If you arrive with a task rather than an optic in mind, start here: - -| I want to… | Recipe | -|-------------------------------------------------------------|--------| -| edit one branch of a sum type, misses passing through | [Visit contingent fields](#visit-contingent-fields) | -| apply one edit at every node of a recursive tree | [Visit across whole trees](#visit-across-whole-trees) | -| update only the matching elements of a collection | [Visit through arbitrary structure](#visit-through-arbitrary-structure) | -| fold several fields into one number | [Isolate what you need](#isolate-what-you-need) | -| rewrite every slot of a fixed shape, or summarise a batch | [Compute aggregations](#compute-aggregations) | -| update a grid coordinate or preserve device labels | [Indexed: Grates and Glasses](indexed.md) | -| change a field deep in JSON without decoding the payload | [Edit JSON without decoding](#edit-json-without-decoding) | -| depend on a relationship instead of a data type | [Require the optic, not the type](#require-the-optic-not-the-type) | -| ask for the weakest capability a function needs | [Depend only on what's needed](#depend-only-on-what-s-needed) | -| edit Avro bytes on a hot path, no decode | [Serdes free pipes](#serdes-free-pipes) | -| move one field between wire formats (Avro ↔ JSON) | [Inspectable cross-format bridges](#inspectable-cross-format-bridges) | -| law-test the optics my module publishes | [Re-usable laws for testing](#re-usable-laws-for-testing) | -| make an in-place update fallible | [Validate-in-place with `modifyF`](#validate-in-place-with-modifyf) | -| swap abort / accumulate / audit without touching the optic | [Structure orthogonal to effects](#structure-orthogonal-to-effects) | -| batch N+1 lookups through one traversal | [Batch-load nested IDs](#batch-load-nested-ids) | -| stamp an effect's result back into the entity | [Persist-and-stamp decode free](#persist-and-stamp-decode-free) | - -```scala mdoc:silent -import dev.constructive.eo.optics.{Lens, Optic, Prism, Traversal} -import dev.constructive.eo.optics.Optic.* -import dev.constructive.eo.data.MultiFocus.given // Functor / Foldable / Traverse for MultiFocus[PSVec] (post-fold) -``` - -## Navigating structures - -One branch of a sum; every node of a recursive tree; a sparse walk -over a collection; several fields at once; and the same moves over -JSON you never decode. The recipes ramp by *shape*, and the point -throughout is that no step dead-ends: whatever you just focused, -the next `.andThen` keeps going. - -### Visit contingent fields - -**Why:** you want to rewrite one variant of a sum type — the bound -variable of a `Var` node in an AST, say — and leave every other -case untouched, without a hand-written `match` that re-builds the -misses. Both hops derive from the type itself: `prism[S, A]` -matches the branch, `lens[S](_.field)` reaches inside it, and one -`.andThen` chains them — no pattern match, no `copy` plumbing, no -optic written by hand. - -```scala mdoc:silent -// The AST under edit — hosted at package level in -// dev.constructive.eo.docs (macro-derived optics need top-level -// targets; mdoc wraps every fence in an object): -// -// enum Expr: -// case Var(name: String) -// case App(f: Expr, x: Expr) -// case Lam(bind: String, body: Expr) -import dev.constructive.eo.docs.Expr -import dev.constructive.eo.generics.{lens, prism} - -val varName = prism[Expr, Expr.Var].andThen(lens[Expr.Var](_.name)) - -// The focus arrives as the case's `(name: String)` NamedTuple — -// the selector covers Var's only field, so the derived optic is a -// full BijectionIso rather than a partial Lens. -val upperVar = varName.modify(v => (name = v.name.toUpperCase)) -``` - -```scala mdoc -// Hit branch: the Var's name is uppercased. -upperVar(Expr.Var("x")) - -// Miss branch: passes through, the Lam is untouched. -upperVar(Expr.Lam("y", Expr.Var("y"))) -``` - -The derived Prism rides the `Either` carrier and the derived field -optic composes across the seam through the same `.andThen` as -everything else on this page. When you need a Prism the macro can't -express — a validating match, a normalising re-build — the -hand-rolled `Prism[S, A](sel, emb)` constructor takes the match -explicitly; see [Generics → `prism[S, A]`](generics.md) for what -the derivation covers (enums, sealed traits, union types) and the -full story on the NamedTuple focus. - -**Source:** Baeldung — *Monocle Optics*, -; framing from -Wlaschin — *Domain Modeling Made Functional*, ch. 4, -. - -### Visit across whole trees - -`varName` edits exactly the node you hand it: `Var("x")` hits, -`Lam(...)` misses — the nested `Var("y")` inside stays untouched, -because a Prism looks at the *top* of the value, not into it. One -more derivation closes the gap: `plate[Expr]` derives the `Plated` -instance (which fields recurse, which are leaves), and -`everywhere` turns it into a recursive optic that reaches every -sub-term. The **same `varName` optic** then composes into it: - -```scala mdoc:silent -import dev.constructive.eo.optics.Plated -import dev.constructive.eo.generics.plate - -given Plated[Expr] = plate[Expr] - -// `everywhere` is `transform` in optic form: everywhere.andThen(d).modify(g) -// applies `d` at every node, bottom-up. Reuse `varName` from above. -val everyVarName = Plated.everywhere[Expr].andThen(varName) - -val upperEveryVar = everyVarName.modify(v => (name = v.name.toUpperCase)) -``` - -```scala mdoc -val term = Expr.App(Expr.Var("f"), Expr.Lam("y", Expr.Var("y"))) - -// varName alone: the top of `term` is an App, not a Var — a miss, -// nothing changes. -upperVar(term) - -// everywhere ∘ varName: every variable at every depth — f -> F and -// the nested y -> Y. The Lam binder "y" (a String, not a Var node) -// is left alone. -upperEveryVar(term) -``` - -That pair of calls is the whole comparison: `varName` is the -surgical single-node edit, `everyVarName` the same edit quantified -over the tree — and the step between them is one derivation plus -one `.andThen`, not a rewrite. The `.modify` runs at every node, -bottom-up and stack-safe to any depth (a million-node tree, or a -100k-deep degenerate spine, won't overflow). When the rewrite is easier -as a plain per-node function, `Plated.transform(f)` is the same engine; -`Plated.universe` / `children` are the read side (every sub-term / -immediate children); `rewrite` repeats an `Expr => Option[Expr]` rule -to a fixpoint. The derivation follows the *exact self-type* rule — -only `Expr`-typed fields are recursion points, so `Lam`'s -`bind: String` stays a leaf. See [Generics → `plate[S]`](generics.md). - -**Source:** Mitchell & Runciman — *Uniform Boilerplate and List -Processing* (Uniplate), -; -Haskell `lens` `Control.Lens.Plated`, -. - -### Visit through arbitrary structure - -**Why:** you have a list of results, some `Ok` and some `Err`, and -want to bump only the successes — leaving the failures exactly -where they are. Walking the container *and* matching the branch in -one pass is the "sparse traversal" that's genuinely annoying to -hand-roll; here it's a one-liner. - -```scala mdoc:silent -import scala.collection.immutable.ArraySeq - -// Result (Ok | Err) is hosted in dev.constructive.eo.docs like Expr -// above — enum macro targets need a package-level home under mdoc. -import dev.constructive.eo.docs.Result - -val okP = prism[Result, Result.Ok] - -val bumpOks = - Traversal.each[ArraySeq, Result] - .andThen(okP) - // Result.Ok is single-field, where the full-cover lens macro returns - // an Iso with a NamedTuple focus — hand-write the Int lens instead. - .andThen(Lens[Result.Ok, Int](_.value, (o, v) => o.copy(value = v))) -``` - -```scala mdoc -bumpOks.modify(_ + 1)( - ArraySeq(Result.Ok(10), Result.Err("nope"), Result.Ok(20)) -) -``` - -Performance: the `PowerSeriesPrismBench` suite measures this -shape at ~5× over a hand-rolled naive loop — and that's the -published worst case for `each`, not the typical one. The -per-benchmark curve lives in -[benchmarks → PowerSeries with Prism inner](benchmarks.md#powerseries-traversal-with-downstream-composition). - -**Source:** Penner — *Optics By Example* ch. 8 (Traversal -Actions) + ch. 10 (Missing Values), -. - -### Isolate what you need - -**Why:** you have a basket of line items and want one number — -total value — without writing a fold by hand or threading two -fields through a loop. The `lens[S](_.a, _.b)` macro focuses both -`quantity` and `price` as a single Scala 3 `NamedTuple`, so the -multiply-and-sum drops straight into `foldMap`. - -```scala mdoc:silent -import cats.instances.list.given // Traverse[List] for .each -import cats.instances.double.given // Monoid[Double] for .foldMap -import dev.constructive.eo.generics.lens - -case class OrderItem(sku: String, quantity: Int, price: Double) - -// Focus both pricing fields at once, then walk every item. -val lineValue = - Traversal.each[List, OrderItem] - .andThen(lens[OrderItem](_.quantity, _.price)) -``` - -```scala mdoc -val inventory = List( - OrderItem("apple", 3, 9.99), - OrderItem("pear", 10, 2.50), - OrderItem("lobster", 2, 45.00), -) - -// One pass: see each item's (quantity, price) NamedTuple, multiply, -// and let the Double monoid sum the line values into a grand total. -lineValue.foldMap(nt => nt.quantity * nt.price)(inventory) -``` - -The focus arrives in selector order, so `nt.quantity` and -`nt.price` are named — the lambda reads like the business rule it -encodes. The same `lineValue` optic also writes (`.modify` to -re-price every line); the fold is just the read-side escape. See -[Generics → Multi-field Lens](generics.md) for the full treatment -of the NamedTuple focus. - -**Source:** cats-eo internal; fold framing from Penner — *Optics -By Example* ch. 7, . - -### Compute aggregations - -A `Lens` sees one value; a `Traversal` visits a container's elements. -`Indexed` addresses a fixed index space, while `MultiFocus[F]` -supports container traversal and aggregation. The two recipes below -show that distinction; see the [MultiFocus reference](multifocus.md). - -#### Recipe A — Adjust a fixed-index configuration - -**Why:** an environment-indexed configuration has a path for each -coordinate. Read one, rewrite each with its index visible, or fill -the whole space without enumerating or comparing indexes. - -```scala mdoc:silent -import cats.Representable -import cats.instances.function.given -import java.nio.file.Path -import dev.constructive.eo.optics.Indexed -import dev.constructive.eo.data.MultiFocus -import dev.constructive.eo.data.MultiFocus.given -import dev.constructive.eo.data.MultiFocus.{collectList, collectMap} - -enum Env: - case Dev, Prod - -val readerRepr = summon[Representable.Aux[[a] =>> Env => a, Env]] -val configGlass = Indexed.representable[[a] =>> Env => a, Path](readerRepr) -val config: Env => Path = - case Env.Prod => Path.of("/opt/app/") - case Env.Dev => Path.of("/dev/app/0292103") -``` - -```scala mdoc -configGlass.at(Env.Dev)(config) -val adjustedConfig = configGlass.modify((env, path) => - if env == Env.Dev then path.resolve("logs") else path -)(config) -(adjustedConfig(Env.Dev), adjustedConfig(Env.Prod)) -val replacePath = Path.of("/") -val replacedConfig = configGlass.replace(replacePath)(config) -(replacedConfig(Env.Dev), replacedConfig(Env.Prod)) -``` - -```scala mdoc:silent -require(configGlass.at(Env.Dev)(config) == Path.of("/dev/app/0292103")) -require(configGlass.at(Env.Prod)(config) == Path.of("/opt/app/")) -require(adjustedConfig(Env.Dev) == config(Env.Dev).resolve("logs")) -require(adjustedConfig(Env.Prod) == config(Env.Prod)) -require(replacedConfig(Env.Dev) == replacePath) -require(replacedConfig(Env.Prod) == replacePath) -``` - -This representable constructor returns `Indexed.Grate`, the -`X = Unit` specialization. Other glasses retain residual context. -Nested `andThen` uses product indexes to address the full grid; -Unit-context normalization and generic classical-family seams remain -open. For effectful per-element rewrites, use container traversal. - -For nested environment/channel settings, type-changing writes, and -labelled sensor grids that retain per-row metadata, continue with -[Indexed: Grates and Glasses](indexed.md). - -**Source:** Penner — *Grate: yet another optic*, -. - -#### Recipe B — Summarise a batch, then broadcast or collapse - -**Why:** a column of readings from a batch needs to become a -*report*: each row pairing the original value with its distance -from the batch average, plus the batch-level statistics — mean, -variance, standard deviation — computed once. The same -`MultiFocus[F]` optic covers every shape the report needs: -broadcast a summary into every position, collapse the batch to a -single footer value, rewrite every row *relative to the batch* in -one expression, or fold out the statistical moments in one pass — -you pick the flavour at the call site: - -```scala mdoc:silent -case class ReportRow(value: Double, distanceFromMean: Double) - -val readings = List(12.0, 14.0, 11.0, 15.0) - -val readingsMF = MultiFocus.apply[List, Double] -``` - -```scala mdoc -// (1) Baseline column — .collectMap broadcasts the batch mean back -// into every position, keeping the shape. -readingsMF.collectMap[Double](xs => xs.sum / xs.size)(readings) - -// (2) Report footer — .collectList collapses the batch to a single -// summary value, whatever the input length. -readingsMF.collectList(xs => xs.sum / xs.size)(readings) - -// (3) The report rows, in one expression — .collectWith is the -// algebraic-lens universal: the aggregate sees the whole batch -// ONCE, computes the baseline, and returns the per-row rewrite. -// The pApply factory makes the walk type-changing, so each -// Double becomes a ReportRow pairing the original with its -// distance from the average. -MultiFocus.pApply[List, Double, ReportRow].collectWith { xs => - val mean = xs.sum / xs.size - v => ReportRow(v, v - mean) -}(readings) - -// (4) The batch-level statistics for the report footer: count, sum -// and sum of squares in ONE foldMap pass via the tuple monoid, -// then the moments fall out. -val (n, sum, sumSq) = readingsMF.foldMap(v => (1, v, v * v))(readings) -val mean = sum / n -val variance = sumSq / n - mean * mean // population variance -val stdDev = math.sqrt(variance) -``` - -**Which flavour?** - -- `.collectMap[B](agg: F[A] => B)` keeps the shape — the summary - lands in every position. Reach for it when a downstream step needs - the aggregate *alongside* the originals (share-of-total, a - normalisation baseline). -- `.collectList(agg: List[A] => B)` supplies a singleton focus vector - to reconstruction with the observed context. This identity-backed - `readingsMF` returns a singleton List, but other optics can preserve - surrounding structure or reject a shape/count mismatch. It is not - a general promise of a single-element source. -- `.collectWith(agg: F[A] => A => B)` is the general map-shaped - collect — the aggregate sees the batch once and the returned - function runs per position, so batch-relative rewrites - (share-of-total, distance-from-mean) are one expression. - `collectMap` is its constant-function special case, and the - type-changing `MultiFocus.pApply[F, A, B]` factory un-pins the - element type so the rewrite can emit a different row type - (step 3). Only `collectList` stays separate — its singleton - collapse changes the focus count, which no map-shaped combinator - can express. - -Why distinct flavours, rather than one derived automatically? The -[MultiFocus reference](multifocus.md#why-two-collect-variants) has -the answer. - -**Sources:** Penner — *Kaleidoscopes: lenses that never die*, -; Penner — -*Algebraic lenses*, . - -### Edit JSON without decoding - -**Why:** you need to change one field deep inside a JSON payload -and pass the rest through untouched — no full decode into a -case-class tree, no re-encode, no decoder for the siblings you -never read. Navigation doesn't stop at the case-class boundary: -`codecPrism[S]` walks circe's `Json` directly, and only the focused -leaf is materialised as `A`: - -```scala mdoc:silent -import dev.constructive.eo.circe.codecPrism -import io.circe.syntax.* - -// The payload's case classes — hosted at package level in -// dev.constructive.eo.docs with kindlings-derived codecs (heavy -// derivation macros run in compiled sources, not mdoc fences): -// -// case class UserAddress(street: String, zip: Int) -// object UserAddress: -// given Codec.AsObject[UserAddress] = KindlingsCodecAsObject.derived -// -// case class SiteUser(name: String, address: UserAddress) // ditto -import dev.constructive.eo.docs.{SiteUser, UserAddress} - -val userStreet = codecPrism[SiteUser].address.street -``` - -```scala mdoc -val userJson = SiteUser("Alice", UserAddress("Main St", 12345)).asJson -userStreet.modifyUnsafe(_.toUpperCase)(userJson).noSpacesSortKeys -``` - -The -[`OrderCirceBench`](https://github.com/Constructive-Programming/eo/blob/main/benchmarks/src/main/scala/dev/constructive/eo/bench/OrderCirceBench.scala) -suite shows this edit is **flat in document size** — ~1.3 µs whether the record -is small or large — while the decode / `.copy` / re-encode path scales with the -whole payload, so the gap grows from ~3× on a tiny record to ~160× on a large -one. circe-optics' analogous `root.user.address.street` surface forces a full -decode per level; cats-eo's cursor walk does not. - -The `*Unsafe` suffix opts out of the failure channel; the default -`modify` returns `Ior[Chain[JsonFailure], Json]`, so a silent -no-op edit tells you which step refused — the -[circe integration's failure flow](integrations/circe.md#failure-flow) -covers routing it. - -**Source:** cats-eo internal (`JsonPrism`); related to circe-optics' -`root.*` idiom . - -#### …and every element of a JSON array - -Walk an array without materialising it as a Scala collection; -only the focused leaf of each element is decoded. The `.each` -step splits the Prism into a `JsonTraversal`: - -```scala mdoc:silent -// Same hosting pattern — kindlings-derived Codec.AsObject on both: -// -// case class Item(name: String, price: Double) -// case class Basket(owner: String, items: Vector[Item]) -import dev.constructive.eo.docs.{Basket, Item} -``` - -```scala mdoc -val basket = Basket("Alice", Vector(Item("apple", 1.0), Item("pear", 2.0))) -val everyItemName = codecPrism[Basket].items.each.name - -everyItemName.modifyUnsafe(_.toUpperCase)(basket.asJson).noSpacesSortKeys -``` - -Per-element failures to decode accumulate into -`Ior.Both(chain, partialJson)` on the default surface — the -[circe integration's failure flow](integrations/circe.md#failure-flow) -shows how to route the chain. - -**Source:** cats-eo internal (`JsonTraversal`). - -## Decoupling and modularity - -Because an optic is a concrete, portable value relating two types, -you can replace a direct dependence on a *data type* with a -dependence on the *relationship*: a function that needs "every -`Instant` in whatever you give me" asks for the optic, not the -type. You can also say precisely how much you need — the guarantees -of an Iso, or just the freedom of a fold — and modules that share -no types at all can still interoperate at the call site, or even at -the wire-format level. - -### Require the optic, not the type - -**Why:** the article's `adjustSheetTimes` shouldn't have to know -about `BalanceSheet`. Any function whose real job is "apply `f` at -every `Instant`" can take the optic as its parameter and let the -caller decide what structure it runs against — the dependence on -the concrete type evaporates: - -```scala mdoc:silent -import java.time.Instant -import dev.constructive.eo.data.{MultiFocus, PSVec} - -def adjustTimes[S](t: Optic[S, S, Instant, Instant, MultiFocus[PSVec]])( - f: Instant => Instant -): S => S = t.modify(f) -``` - -Any structure that can point at its `Instant`s gets the behaviour — -all it takes is that structure's own optic. A record whose -timestamps sit in two fields, and a bare collection, are served by -the same function: - -```scala mdoc:silent -case class Meeting(subject: String, start: Instant, end: Instant) - -val meetingTimes = - Lens[Meeting, (Instant, Instant)]( - m => (m.start, m.end), - (m, se) => m.copy(start = se._1, end = se._2), - ).andThen(Traversal.two[(Instant, Instant), (Instant, Instant), Instant, Instant]( - _._1, - _._2, - (_, _), - )) - -val bumpMeeting = adjustTimes(meetingTimes)(_.plusSeconds(3600)) -val bumpAll = adjustTimes(Traversal.each[List, Instant])(_.plusSeconds(3600)) -``` - -```scala mdoc -bumpMeeting(Meeting("standup", Instant.EPOCH, Instant.EPOCH.plusSeconds(900))) -bumpAll(List(Instant.EPOCH, Instant.EPOCH.plusSeconds(60))) -``` - -`Meeting` and `List[Instant]` share no interface, no parent trait, -no typeclass — the optic *is* the interface. This is the article's -central move: interoperability between otherwise wholly independent -modules, done at the call site, with the structure-owning module -publishing one optic value instead of exposing its shape. - -**Source:** Hansen — *We Need More Optics*, The Startup, - -("Modular Application Design"). - -### Re-use an optic across representations - -**Why:** two subsystems rarely agree on representation — one thinks -in UTC `Instant`s, another in wall-clock `LocalDateTime`s; one holds -the domain record, another the wire tuple it serialises to. An optic -is a value relating two types, and `Optic` carries two lawful -`cats.arrow.Profunctor` instances that let you *re-aim* a published -optic instead of re-deriving it: `Optic.innerProfunctor` maps the -focus pair (what the optic yields, and what it accepts back), and -`Optic.outerProfunctor` maps the source pair (what it reads from, -and what it rebuilds). - -Re-aim the focus — `meetingTimes` above yields `Instant`s, but the -calendar subsystem edits wall-clock times: - -```scala mdoc:silent -import java.time.{LocalDateTime, ZoneOffset} - -val meetingWallClock = - Optic - .innerProfunctor[Meeting, Meeting, MultiFocus[PSVec]] - .dimap(meetingTimes)((wall: LocalDateTime) => wall.toInstant(ZoneOffset.UTC))( - LocalDateTime.ofInstant(_, ZoneOffset.UTC) - ) -``` - -```scala mdoc -meetingWallClock.modify(_.plusHours(2))( - Meeting("standup", Instant.EPOCH, Instant.EPOCH.plusSeconds(900)) -) -``` - -Re-aim the source — the scheduling gateway never sees `Meeting`, only -the `(startEpochSecond, endEpochSecond)` pair it puts on the wire. -The same optic serves it too: - -```scala mdoc:silent -val wireTimes = - Optic - .outerProfunctor[Instant, Instant, MultiFocus[PSVec]] - .dimap(meetingTimes)((p: (Long, Long)) => - Meeting("wire", Instant.ofEpochSecond(p._1), Instant.ofEpochSecond(p._2)) - )(m => (m.start.getEpochSecond, m.end.getEpochSecond)) -``` - -```scala mdoc -wireTimes.modify(_.plusSeconds(3600))((0L, 900L)) -``` - -The same moves cover any representation split: UTF-8 vs UTF-16 text -(dimap through the re-encoding pair), big- vs little-endian words -(dimap through `java.lang.Long.reverseBytes`), or a legacy envelope -around the record you actually care about. - -One caveat: `dimap` returns an anonymous `Optic`, so the generic -capability surface (`.modify`, `.foldMap`, …) applies but -concrete-class fused overloads are erased. When the concrete optic -ships its own input-side mapping — `Prism.tearFrom` / `mendFrom` — -prefer that form on hot paths. - -### Depend only on what's needed - -**Why:** the recipe above still pins the carrier — it accepts -traversals only, and the optic travels as an explicit argument. -Most functions need even less: "something I can fold `Double`s out -of", "something I can rewrite `Double`s through". The -[capability traits](capabilities.md) — `CanGet`, `CanGetOption`, -`CanModify`, `CanFold`, … — are exactly those contracts as proper -types, so the optic arrives as **`using` evidence** and the subject -type stays fully generic. This is the article's -`implicit T: Traversal[T, DateTime]` move, with the carrier erased: - -```scala mdoc:silent -import dev.constructive.eo.{CanFold, CanModify} - -// Read-side contract: anything that can fold Doubles out of S — -// Lens, Prism, Optional, Traversal, Fold, Iso all qualify. -def total[S](s: S)(using o: CanFold[S, Double]): Double = - o.foldMap(identity)(s) - -// Write-side contract: anything that can rewrite Doubles in place. -def scale[S](k: Double)(using cm: CanModify[S, Double]): S => S = - cm.modify(_ * k) -``` - -```scala mdoc:silent -case class Line(desc: String, amount: Double) -case class Invoice(fee: Double, lines: List[Line]) - -val feeL = lens[Invoice](_.fee) - -val lineAmounts = - lens[Invoice](_.lines) - .andThen(Traversal.each[List, Line]) - .andThen(lens[Line](_.amount)) -``` - -Concrete optic classes *implement* the capabilities, so a lens can -be handed over as the evidence itself; an optic known only at the -generic `Optic[…, F]` type — like the composed traversal — is bound -as a `given` and the capability is derived on the spot: - -```scala mdoc -val inv = Invoice(5.0, List(Line("widgets", 10.0), Line("gadgets", 20.0))) - -// A Lens IS a CanFold — pass it as the evidence directly. -total(inv)(using feeL) - -// The composed traversal is an anonymous Optic; bind it as a given -// and both contracts derive from it. -given Optic[Invoice, Invoice, Double, Double, MultiFocus[PSVec]] = lineAmounts -total(inv) -scale(1.1)(inv) -``` - -The trait ladder, weakest first: `CanFold` (`foldMap` / -`headOption` / `exists` / `length` / `foci`), `CanGetOption`, -`CanModify` (`modify` / `replace`), `CanGet`, `CanReverseGet` -(build). A signature that demands only `CanFold` accepts -everything; one that demands both `CanGet` and `CanReverseGet` -insists on an Iso. Under the hood each trait is gated by the -matching typeclass on the carrier (`ForgetfulFold[F]`, -`PartialAccessor[F]`, `ForgetfulFunctor[F]`, `Accessor[F]`, -`ReverseAccessor[F]`) — a signature can still take `Optic[…, F]` -plus the gate directly when it genuinely needs the carrier; see -[Capabilities](capabilities.md) for the full matrix, the coherence -rules, and the using-clause ordering footgun that page documents -for hand-written gate signatures. - -**Source:** Hansen — *We Need More Optics* ("We can eliminate the -dependence on the BalanceSheet type by requiring a Traversal -instead"; "you are able to specify if you need the guarantees of an -Iso, or the freedom of a Getter"). - -### Serdes free pipes - -**Why:** a Kafka consumer gets an `Array[Byte]` payload, needs to -change one field, and re-emits binary on the producer side. The -conventional answer couples the consumer to the *whole* schema — a -decode into the full case-class tree per message, on a hot path -where that is pure overhead. `AvroPrism` decouples it: the module -depends on one field's optic, not on `OrderEvent`. It IS a -byte-carried optic (`Optic[Array[Byte], Array[Byte], A, A, -Affine]`): it locates the focused field's byte span under a cached -reader schema, decodes only that slice, and writes by splicing the -re-encoded focus back into the payload — no `IndexedRecord` -materialised on either side: - -```scala mdoc:silent -import dev.constructive.eo.avro as eoavro -import dev.constructive.eo.avro.AvroCodec -import java.io.ByteArrayOutputStream -import org.apache.avro.generic.{GenericDatumWriter, GenericRecord} -import org.apache.avro.io.EncoderFactory - -// Hosted in dev.constructive.eo.docs with kindlings-derived -// AvroEncoder / AvroDecoder / AvroSchemaFor givens: -// -// case class OrderEvent(orderId: String, customer: String, total: Double) -import dev.constructive.eo.docs.OrderEvent - -// Stand-in for an inbound Kafka record: serialise an OrderEvent to -// binary under the same schema the prism caches. -val outSchema = summon[AvroCodec[OrderEvent]].schema -val sample = OrderEvent("ord-42", "alice", 99.99) -val sampleBytes: Array[Byte] = - val rec = summon[AvroCodec[OrderEvent]].encode(sample).asInstanceOf[GenericRecord] - val out = new ByteArrayOutputStream() - val encoder = EncoderFactory.get().binaryEncoder(out, null) - val writer = new GenericDatumWriter[GenericRecord](outSchema) - writer.write(rec, encoder) - encoder.flush() - out.toByteArray - -// The optic: walk into `customer` once at construction time, -// reuse on every inbound record. Disambiguate the Avro -// `codecPrism` from the circe one imported earlier in this page -// by qualifying through the `eoavro` alias. -val upperCustomer = eoavro.codecPrism[OrderEvent].customer -``` - -```scala mdoc -// Kafka hot path: bytes in → splice in place → bytes out. No -// re-serialisation step — the prism's write side IS the emit path. -val outBytes: Array[Byte] = - upperCustomer.modify(_.toUpperCase)(sampleBytes) - -// Round-trip witness — decode the output to confirm the customer -// field changed and the rest is preserved. -(upperCustomer.getOption(outBytes), - eoavro.codecPrism[OrderEvent].getOption(outBytes)) -``` - -`.modify` on the byte optic is silent-pass-through — bad bytes, -missing fields, or decode mismatches leave the input payload -untouched (an `Affine` Miss) rather than allocating an `Ior` -chain. That matches the Kafka consumer budget: at-least-once -delivery already implies the consumer must be tolerant of -malformed payloads at the offset commit boundary, and the -per-record allocation cost of `Ior` is a tax on the happy path. -When you DO want the diagnostic — for a dead-letter queue, say — -flip to the record-carried face: -`upperCustomer.record.modify(...)` accepts the same bytes and -returns `Ior[Chain[AvroFailure], IndexedRecord]`; route on the -`Ior.Both` / `Ior.Left` shape from there. - -The cached reader schema is the load-bearing piece: a single -`codecPrism[OrderEvent]` value pins the schema once, and the -parser reuses it across millions of inbound records. For the -schema-registry case where the reader schema arrives at -runtime, use the explicit-schema overload — -`AvroPrism.codecPrism[OrderEvent](runtimeSchema)` — to bypass -the kindlings-derived schema entirely. - -**Source:** cats-eo internal (`AvroPrism`'s byte-carried -default surface). Background framing on the streaming / -Kafka use case lives in the -[Avro integration intro](integrations/avro.md#why-this-exists). - -### Inspectable cross-format bridges - -**Why:** the consumer reads Avro off a topic and must emit JSON -to a partner API — or receives JSON and must re-emit Avro. Two -modules, two wire formats, and the only thing they genuinely share -is one field. The conventional bridge couples them anyway: decode -the whole record into a case class, build a whole output value, -encode it all. With a byte optic on each side of the seam, only the -moved branch is ever decoded and re-encoded: the full object is -**never constructed** on either format. (`AvroJsonBridgeSpec` -proves this with a counting root codec — both bridge directions run -with zero root decodes and zero root encodes; `AvroJsonBridgeBench` -puts B/op numbers on it.) - -```scala mdoc:silent -import com.github.plokhotnyuk.jsoniter_scala.core.JsonValueCodec -import com.github.plokhotnyuk.jsoniter_scala.macros.JsonCodecMaker -import dev.constructive.eo.jsoniter.JsoniterPrism - -given JsonValueCodec[String] = JsonCodecMaker.make - -// One byte optic per format, same focus type. Drilled once, -// reused for every message. (`JsoniterPrism.fromPath` returns an -// `Either` — a path is data; the literal below is unwrapped here.) -val customerAvro = eoavro.codecPrism[OrderEvent].customer -val customerJson = - JsoniterPrism - .fromPath[String]("$.customer") - .fold(msg => throw new IllegalArgumentException(msg), identity) - -// The JSON side's output skeleton. Placeholders must be VALID -// encodings of the branch type — the splice write decodes the -// current focus before replacing it. -val receiptTemplate: Array[Byte] = - """{"kind":"receipt","customer":"","status":"ok"}""".getBytes("UTF-8") -``` - -```scala mdoc -// Avro → JSON: slice the branch off the Avro wire form, splice -// it into the JSON template. No OrderEvent, no receipt object. -val receipt = customerAvro.getOption(sampleBytes) match - case Some(c) => customerJson.replace(c)(receiptTemplate) - case None => receiptTemplate -new String(receipt, "UTF-8") -``` - -```scala mdoc -// JSON → Avro: read the branch from JSON bytes, splice it into -// existing Avro wire bytes. Same guarantee, reversed. -val inbound: Array[Byte] = - """{"kind":"receipt","customer":"carol","status":"ok"}""".getBytes("UTF-8") -val updatedAvro = customerJson.getOption(inbound) match - case Some(c) => customerAvro.replace(c)(sampleBytes) - case None => sampleBytes - -// Witness (this read DOES construct the object — the bridge -// above never did): the branch moved, the siblings survived. -eoavro.codecPrism[OrderEvent].getOption(updatedAvro) -``` - -Cost model worth knowing: each `.replace` locates the span, -decodes the *current* focus (the `Affine` write carries it), -re-encodes the new one, and allocates one output buffer. Moving -one branch this way is far cheaper than materialising a wide -object — but for many branches per message, or fragment moves -between two *Avro* payloads, prefer `sliceBytes` / `graftBytes` -(no decode at all). - -**Source:** `AvroJsonBridgeSpec` (avro module, `eo.avro.jsoniter`) and -`AvroJsonBridgeBench` (benchmarks) in cats-eo. - -### Re-usable laws for testing - -A decoupled boundary needs a tested contract. Composing optics -means the combinators are already tested — what's left to check is -*your* instances, the optic values your module publishes. The -discipline suites that gate cats-eo's own carriers ship in -`cats-eo-laws`, so a composed optic like `meetingTimes` gets the -full Traversal rule-set in one `checkAll` instead of hand-written -round-trip tests: - -```scala -libraryDependencies += "dev.constructive" %% "cats-eo-laws" % "@VERSION@" % Test -``` - -```scala -import cats.Functor -import dev.constructive.eo.laws.TraversalLaws -import dev.constructive.eo.laws.discipline.TraversalTests - -// meetingTimes walks a plain Meeting, so T is the constant type -// lambda — its Functor is one line. Arbitrary[Meeting] and -// Cogen[Instant] come from your ScalaCheck generators. -given Functor[[X] =>> Meeting] with - def map[A, B](fa: Meeting)(f: A => B): Meeting = fa - -checkAll( - "meetingTimes", - new TraversalTests[[X] =>> Meeting, Instant]: - val laws = new TraversalLaws[[X] =>> Meeting, Instant]: - val traversal = meetingTimes - .traversal, -) -``` - -Every optic family has a matching `FooLaws` / `FooTests` pair — the -[migration guide](migration-from-monocle.md#what-stays-the-same) has -the import-swap table. - -**Source:** Hansen — *We Need More Optics*, The Startup, - ("Re-usable -tests"). - -## Effect threading - -Optics are great at turning a host language's control flow into -just type definitions. The optic fixes *where* — which foci, at -what depth; the effect type `G` you thread through `.modifyF` / -`.modifyA` decides *what the control flow is*: abort on first -failure, accumulate every failure, log alongside the rewrite, defer -into `IO`. Swapping behaviours means swapping `G`, never touching -the optic. - -### Validate-in-place with `modifyF` - -**Why:** the update can fail. Bump `age` by 1, but reject a -negative input with `None` — and keep the validation and the write -as one expression instead of a get / check / set dance. `.modifyF[G]` -lifts an `A => G[B]` through any carrier that admits `Functor[G]`: - -```scala mdoc:silent -case class Visitor(name: String, age: Int) - -val visitorAgeL = lens[Visitor](_.age) -``` - -```scala mdoc -import cats.syntax.functor.* -import cats.instances.option.* - -visitorAgeL.modifyF[Option](age => - if age >= 0 then Some(age + 1) else None -)(Visitor("Alice", 30)) - -visitorAgeL.modifyF[Option](age => - if age >= 0 then Some(age + 1) else None -)(Visitor("Alice", -1)) -``` - -`.modifyA[G]` is the `Applicative[G]` variant — reach for it -when the chain has branching effects to combine (traversal + -validation, for instance); the next recipe is exactly that. -`Witherable`-style filter-and-drop traversal is cross-referenced from -[Penner — *Composable filters using Witherable optics*](https://chrispenner.ca/posts/witherable-optics); -the carrier is deferred to a follow-up release. - -**Source:** Monocle / Haskell `lens` classic `traverseOf`, -generalised. - -### Structure orthogonal to effects - -**Why:** the article's `systemVerifier` insight — "what to do is -decided by the effect type" — in its smallest honest form. Take the -`lineAmounts` traversal from the "Depend only on what's needed" -recipe and thread three different `G`s through the *same* -`.modifyA`. The -optic never changes; the control flow does: - -```scala mdoc:silent -import cats.data.{ValidatedNel, Writer} -import cats.syntax.validated.* - -type Checked[A] = ValidatedNel[String, A] -type Audit[A] = Writer[List[String], A] - -val badInv = Invoice(5.0, List(Line("widgets", -10.0), Line("gadgets", -20.0))) -``` - -```scala mdoc -// G = Option: abort on the first bad focus — short-circuit control flow. -lineAmounts.modifyA[Option](a => - if a > 0 then Some(a * 1.1) else None -)(badInv) - -// G = ValidatedNel: visit EVERY focus, accumulate every failure — -// same optic, same lambda shape, completely different control flow. -lineAmounts.modifyA[Checked](a => - if a > 0 then (a * 1.1).validNel else s"non-positive amount: $a".invalidNel -)(badInv) - -// G = Writer: rewrite AND emit an audit line per focus — the effect -// threads a log through the walk without touching the domain types. -lineAmounts.modifyA[Audit](a => - Writer(List(s"scaled $a"), a * 1.1) -)(inv).run -``` - -The same slot takes `IO` (each focus becomes an effect to -sequence), `Eval` (defer the whole rewrite), or an fs2 `Stream` — -the article's original `systemVerifier` pipes every focus of a -traversal through a verification stream with `.modifyF`, and its -shape is exactly the three calls above with a fancier `G`. The -domain types never learn any of this is happening. - -**Source:** Hansen — *We Need More Optics*, The Startup, - ("What -to do is decided by the effect type"). - -### Batch-load nested IDs - -**Why:** every node in a structure carries an ID you need to -resolve against a database, and the naive `.modifyA` fires one -query per node — the classic N+1. Use the same traversal twice: -`foldMap` collects every ID in one pass, you issue a single batched -query, then `.modify` distributes the results back. 100×–300× -fewer queries with no change to the domain types: - -```scala mdoc:silent -case class Node(id: Int, label: String) -case class Payload(id: Int, body: String) - -// Stand-in for a DB-backed batch fetch. -def fetchAll(ids: List[Int]): Map[Int, Payload] = - ids.map(i => i -> Payload(i, s"body-$i")).toMap - -val eachLeaf = Traversal.each[List, Node] -``` - -```scala mdoc -val nodes = List(Node(1, "a"), Node(2, "b"), Node(3, "c")) - -// Pass 1: collect every ID into a single list via foldMap. -val allIds = eachLeaf.foldMap((n: Node) => List(n.id))(nodes) - -// Pass 2: issue ONE query for the whole set. -val byId = fetchAll(allIds) - -// Pass 3: broadcast the resolved payloads back through the same -// traversal. The structure is preserved; only the labels shift. -eachLeaf.modify(n => n.copy(label = byId(n.id).body))(nodes) -``` - -The pattern generalises to trees, graphs, nested containers, -anything with a `Traverse`. The two-pass idiom is what cats-eo's -`foldMap` + `modify` pair already enables out of the box — no -cats-eo-specific API to learn. - -**Source:** Penner — *Using traversals to batch database -queries*, . - -### Persist-and-stamp decode free - -**Why:** a `PUT` (or `POST`) handler receives a draft entity on the -wire, persists it, and must return the *same* entity enriched with -the database-assigned id. The id only exists after the effectful -store runs, so this is the classic "set one field to the result of -an effect" shape — and a derived id-lens makes the stamp a one-liner. -With circe carrying the value across the wire on both ends, the -whole handler is a short pipeline: `decode → store → stamp → -encode`. - -```scala mdoc:silent -import io.circe.Json -import io.circe.syntax.* -import io.circe.parser.decode -import dev.constructive.eo.generics.lens - -// cats.Eval stands in for your effect type (cats-effect IO, ZIO, -// Future…) — deferred, has map/flatMap, already on the classpath. -import cats.Eval - -// Hosted in dev.constructive.eo.docs with a kindlings-derived -// Codec.AsObject given: -// -// final case class BalanceSheet(id: Long, owner: String, total: Double) -import dev.constructive.eo.docs.BalanceSheet - -// One derived lens onto the id — no hand-written getter/setter, and -// it works even though `id` shares the case class with two other fields. -val sheetId = lens[BalanceSheet](_.id) - -// The effectful store: inserts the row, hands back the generated id. -def save(sheet: BalanceSheet): Eval[Long] = Eval.always { - val _ = sheet // pretend: INSERT … RETURNING id - 42L -} - -// Persist, then stamp the returned id back onto the object with the -// lens — `save(...).map(sheetId.replace(_)(sheet))`. -def store(sheet: BalanceSheet): Eval[BalanceSheet] = - save(sheet).map(sheetId.replace(_)(sheet)) - -// The whole PUT handler: request bytes in, response bytes out. -def handlePut(body: String): Eval[String] = - decode[BalanceSheet](body) match - case Right(draft) => store(draft).map(_.asJson.noSpaces) - case Left(err) => Eval.now(Json.obj("error" -> err.getMessage.asJson).noSpaces) -``` - -```scala mdoc -// Incoming PUT body — `id` is a placeholder the store overwrites. -val request = """{"id":0,"owner":"Acme Corp","total":1234.5}""" - -handlePut(request).value -``` - -Every step earns its place: `decode` is the wire → domain hop -(circe), `store` is your effect, the stamp is the derived lens -standing in for a hand-written `sheet.copy(id = newId)`, and -`asJson` is the domain → wire hop back out. The lens is the only -optic, and it stays a reusable value — compose it further -(`orderLens.andThen(sheetId)`) the moment the entity nests inside a -larger request. In production you'd reach for an opaque `DbId` -rather than a bare `Long`, and likely a separate `Draft` type -without the id field; the shape of the pipeline doesn't change. - -**Source:** the lens-stamps-the-effect-result pattern, -; -see also [Generics → Composing into pipelines](generics.md#composing-derived-optics-into-pipelines). - -## Further reading - -- [Concepts](concepts.md) — the theory behind the unified Optic - trait and carriers; the - [composition lattice](concepts.md#composition-lattice) - diagram maps out every `Composer[F, G]` bridge. -- [Optics reference](optics.md) — the full per-family tour, - introduced by the - [family taxonomy](optics.md#family-taxonomy) diagram. -- [Generics](generics.md) — macro-derived `lens[S](...)` and - `prism[S, A]`. -- [Circe integration](integrations/circe.md) — cursor-backed JSON optics; - [failure flow](integrations/circe.md#failure-flow) for the Ior decision - tree. -- [Avro integration](integrations/avro.md) — cursor-backed Avro optics; - [failure flow](integrations/avro.md#failure-flow) for the schema-driven - Ior decision tree. -- [Migrating from Monocle](migration-from-monocle.md) — - side-by-side translation guide. -- Hansen — *We Need More Optics*, The Startup, - — the - article this page's three-part structure follows. diff --git a/site/docs/cookbook/README.md b/site/docs/cookbook/README.md new file mode 100644 index 00000000..9fcb59a1 --- /dev/null +++ b/site/docs/cookbook/README.md @@ -0,0 +1,81 @@ +# Cookbook + +Optics are probably the most powerful unused tool in our arsenal — +this section is the working proof. Runnable patterns for the questions +that come up most often, one page for each of the three jobs optics do +best: + +1. **[Navigating structures](navigating-structures.md)** — reach one + branch of a sum, every node of a recursive tree, several fields + at once, or a field inside bytes you never decode. One `.andThen` + surface for all of it. +2. **[Decoupling and modularity](decoupling.md)** — + replace a direct dependence on a data type with an optic: depend + on the *relationship*, ask for the *weakest capability* you need, + and let independent modules interoperate at the call site. +3. **[Effect threading](effect-threading.md)** — turn control flow + into type definitions: the optic pins *where*, the effect type + decides *what happens* — abort, accumulate, audit, defer. + +Two demos open the section: whole programs, with their own tests and +benchmarks, that put these recipes to work. + +- **[Allocation-free stream triage](grpc-filter.md)** — an fs2 filter + that classifies and filters a gRPC stream of 64-field JSON messages + straight off a socket, reading the few values it needs with + eo-jsoniter instead of decoding each message. +- **[Blockchain smart contract compilation](uplc-jvm.md)** — a compiler + from Cardano's Plutus Core to JVM class files, built as a chain of + lawful prisms and an iso, and checked against the reference + evaluator on every program. + +The framing follows Hansen — *We Need More Optics* +() and +Penner — *Optics By Example* +(). These are worked +examples, not a tutorial — for the optic families themselves see the +[Optics reference](../optics.md). Every recipe cites its source. + +If you arrive with a task rather than an optic in mind, start here: + +| I want to… | Recipe | +|-------------------------------------------------------------|--------| +| edit one branch of a sum type, misses passing through | [Visit contingent fields](navigating-structures.md#visit-contingent-fields) | +| apply one edit at every node of a recursive tree | [Visit across whole trees](navigating-structures.md#visit-across-whole-trees) | +| update only the matching elements of a collection | [Visit through arbitrary structure](navigating-structures.md#visit-through-arbitrary-structure) | +| fold several fields into one number | [Isolate what you need](navigating-structures.md#isolate-what-you-need) | +| rewrite every slot of a fixed shape, or summarise a batch | [Compute aggregations](navigating-structures.md#compute-aggregations) | +| update a grid coordinate or preserve device labels | [Indexed: Grates and Glasses](../indexed.md) | +| change a field deep in JSON without decoding the payload | [Edit JSON without decoding](navigating-structures.md#edit-json-without-decoding) | +| depend on a relationship instead of a data type | [Require the optic, not the type](decoupling.md#require-the-optic-not-the-type) | +| ask for the weakest capability a function needs | [Depend only on what's needed](decoupling.md#depend-only-on-what-s-needed) | +| edit Avro bytes on a hot path, no decode | [Serdes free pipes](decoupling.md#serdes-free-pipes) | +| move one field between wire formats (Avro ↔ JSON) | [Inspectable cross-format bridges](decoupling.md#inspectable-cross-format-bridges) | +| law-test the optics my module publishes | [Re-usable laws for testing](decoupling.md#re-usable-laws-for-testing) | +| make an in-place update fallible | [Validate-in-place with `modifyF`](effect-threading.md#validate-in-place-with-modifyf) | +| swap abort / accumulate / audit without touching the optic | [Structure orthogonal to effects](effect-threading.md#structure-orthogonal-to-effects) | +| batch N+1 lookups through one traversal | [Batch-load nested IDs](effect-threading.md#batch-load-nested-ids) | +| stamp an effect's result back into the entity | [Persist-and-stamp decode free](effect-threading.md#persist-and-stamp-decode-free) | + +## Further reading + +- [Concepts](../concepts.md) — the theory behind the unified Optic + trait and carriers; the + [composition lattice](../concepts.md#composition-lattice) + diagram maps out every `Composer[F, G]` bridge. +- [Optics reference](../optics.md) — the full per-family tour, + introduced by the + [family taxonomy](../optics.md#family-taxonomy) diagram. +- [Generics](../generics.md) — macro-derived `lens[S](...)` and + `prism[S, A]`. +- [Circe integration](../integrations/circe.md) — cursor-backed JSON optics; + [failure flow](../integrations/circe.md#failure-flow) for the Ior decision + tree. +- [Avro integration](../integrations/avro.md) — cursor-backed Avro optics; + [failure flow](../integrations/avro.md#failure-flow) for the schema-driven + Ior decision tree. +- [Migrating from Monocle](../migration-from-monocle.md) — + side-by-side translation guide. +- Hansen — *We Need More Optics*, The Startup, + — the + article this section's three pages follow. diff --git a/site/docs/cookbook/decoupling.md b/site/docs/cookbook/decoupling.md new file mode 100644 index 00000000..8aff60b3 --- /dev/null +++ b/site/docs/cookbook/decoupling.md @@ -0,0 +1,403 @@ +# Decoupling and modularity + +Because an optic is a concrete, portable value relating two types, +you can replace a direct dependence on a *data type* with a +dependence on the *relationship*: a function that needs "every +`Instant` in whatever you give me" asks for the optic, not the +type. You can also say precisely how much you need — the guarantees +of an Iso, or just the freedom of a fold — and modules that share +no types at all can still interoperate at the call site, or even at +the wire-format level. + +```scala mdoc:silent +import dev.constructive.eo.optics.{Lens, Optic, Prism, Traversal} +import dev.constructive.eo.optics.Optic.* +import dev.constructive.eo.data.MultiFocus.given // Functor / Foldable / Traverse for MultiFocus[PSVec] (post-fold) +``` + +## Require the optic, not the type + +**Why:** the article's `adjustSheetTimes` shouldn't have to know +about `BalanceSheet`. Any function whose real job is "apply `f` at +every `Instant`" can take the optic as its parameter and let the +caller decide what structure it runs against — the dependence on +the concrete type evaporates: + +```scala mdoc:silent +import java.time.Instant +import dev.constructive.eo.data.{MultiFocus, PSVec} + +def adjustTimes[S](t: Optic[S, S, Instant, Instant, MultiFocus[PSVec]])( + f: Instant => Instant +): S => S = t.modify(f) +``` + +Any structure that can point at its `Instant`s gets the behaviour — +all it takes is that structure's own optic. A record whose +timestamps sit in two fields, and a bare collection, are served by +the same function: + +```scala mdoc:silent +case class Meeting(subject: String, start: Instant, end: Instant) + +val meetingTimes = + Lens[Meeting, (Instant, Instant)]( + m => (m.start, m.end), + (m, se) => m.copy(start = se._1, end = se._2), + ).andThen(Traversal.two[(Instant, Instant), (Instant, Instant), Instant, Instant]( + _._1, + _._2, + (_, _), + )) + +val bumpMeeting = adjustTimes(meetingTimes)(_.plusSeconds(3600)) +val bumpAll = adjustTimes(Traversal.each[List, Instant])(_.plusSeconds(3600)) +``` + +```scala mdoc +bumpMeeting(Meeting("standup", Instant.EPOCH, Instant.EPOCH.plusSeconds(900))) +bumpAll(List(Instant.EPOCH, Instant.EPOCH.plusSeconds(60))) +``` + +`Meeting` and `List[Instant]` share no interface, no parent trait, +no typeclass — the optic *is* the interface. This is the article's +central move: interoperability between otherwise wholly independent +modules, done at the call site, with the structure-owning module +publishing one optic value instead of exposing its shape. + +**Source:** Hansen — *We Need More Optics*, The Startup, + +("Modular Application Design"). + +## Re-use an optic across representations + +**Why:** two subsystems rarely agree on representation — one thinks +in UTC `Instant`s, another in wall-clock `LocalDateTime`s; one holds +the domain record, another the wire tuple it serialises to. An optic +is a value relating two types, and `Optic` carries two lawful +`cats.arrow.Profunctor` instances that let you *re-aim* a published +optic instead of re-deriving it: `Optic.innerProfunctor` maps the +focus pair (what the optic yields, and what it accepts back), and +`Optic.outerProfunctor` maps the source pair (what it reads from, +and what it rebuilds). + +Re-aim the focus — `meetingTimes` above yields `Instant`s, but the +calendar subsystem edits wall-clock times: + +```scala mdoc:silent +import java.time.{LocalDateTime, ZoneOffset} + +val meetingWallClock = + Optic + .innerProfunctor[Meeting, Meeting, MultiFocus[PSVec]] + .dimap(meetingTimes)((wall: LocalDateTime) => wall.toInstant(ZoneOffset.UTC))( + LocalDateTime.ofInstant(_, ZoneOffset.UTC) + ) +``` + +```scala mdoc +meetingWallClock.modify(_.plusHours(2))( + Meeting("standup", Instant.EPOCH, Instant.EPOCH.plusSeconds(900)) +) +``` + +Re-aim the source — the scheduling gateway never sees `Meeting`, only +the `(startEpochSecond, endEpochSecond)` pair it puts on the wire. +The same optic serves it too: + +```scala mdoc:silent +val wireTimes = + Optic + .outerProfunctor[Instant, Instant, MultiFocus[PSVec]] + .dimap(meetingTimes)((p: (Long, Long)) => + Meeting("wire", Instant.ofEpochSecond(p._1), Instant.ofEpochSecond(p._2)) + )(m => (m.start.getEpochSecond, m.end.getEpochSecond)) +``` + +```scala mdoc +wireTimes.modify(_.plusSeconds(3600))((0L, 900L)) +``` + +The same moves cover any representation split: UTF-8 vs UTF-16 text +(dimap through the re-encoding pair), big- vs little-endian words +(dimap through `java.lang.Long.reverseBytes`), or a legacy envelope +around the record you actually care about. + +One caveat: `dimap` returns an anonymous `Optic`, so the generic +capability surface (`.modify`, `.foldMap`, …) applies but +concrete-class fused overloads are erased. When the concrete optic +ships its own input-side mapping — `Prism.tearFrom` / `mendFrom` — +prefer that form on hot paths. + +## Depend only on what's needed + +**Why:** the recipe above still pins the carrier — it accepts +traversals only, and the optic travels as an explicit argument. +Most functions need even less: "something I can fold `Double`s out +of", "something I can rewrite `Double`s through". The +[capability traits](../capabilities.md) — `CanGet`, `CanGetOption`, +`CanModify`, `CanFold`, … — are exactly those contracts as proper +types, so the optic arrives as **`using` evidence** and the subject +type stays fully generic. This is the article's +`implicit T: Traversal[T, DateTime]` move, with the carrier erased: + +```scala mdoc:silent +import dev.constructive.eo.{CanFold, CanModify} + +// Read-side contract: anything that can fold Doubles out of S — +// Lens, Prism, Optional, Traversal, Fold, Iso all qualify. +def total[S](s: S)(using o: CanFold[S, Double]): Double = + o.foldMap(identity)(s) + +// Write-side contract: anything that can rewrite Doubles in place. +def scale[S](k: Double)(using cm: CanModify[S, Double]): S => S = + cm.modify(_ * k) +``` + +```scala mdoc:silent +import dev.constructive.eo.generics.lens + +case class Line(desc: String, amount: Double) +case class Invoice(fee: Double, lines: List[Line]) + +val feeL = lens[Invoice](_.fee) + +val lineAmounts = + lens[Invoice](_.lines) + .andThen(Traversal.each[List, Line]) + .andThen(lens[Line](_.amount)) +``` + +Concrete optic classes *implement* the capabilities, so a lens can +be handed over as the evidence itself; an optic known only at the +generic `Optic[…, F]` type — like the composed traversal — is bound +as a `given` and the capability is derived on the spot: + +```scala mdoc +val inv = Invoice(5.0, List(Line("widgets", 10.0), Line("gadgets", 20.0))) + +// A Lens IS a CanFold — pass it as the evidence directly. +total(inv)(using feeL) + +// The composed traversal is an anonymous Optic; bind it as a given +// and both contracts derive from it. +given Optic[Invoice, Invoice, Double, Double, MultiFocus[PSVec]] = lineAmounts +total(inv) +scale(1.1)(inv) +``` + +The trait ladder, weakest first: `CanFold` (`foldMap` / +`headOption` / `exists` / `length` / `foci`), `CanGetOption`, +`CanModify` (`modify` / `replace`), `CanGet`, `CanReverseGet` +(build). A signature that demands only `CanFold` accepts +everything; one that demands both `CanGet` and `CanReverseGet` +insists on an Iso. Under the hood each trait is gated by the +matching typeclass on the carrier (`ForgetfulFold[F]`, +`PartialAccessor[F]`, `ForgetfulFunctor[F]`, `Accessor[F]`, +`ReverseAccessor[F]`) — a signature can still take `Optic[…, F]` +plus the gate directly when it genuinely needs the carrier; see +[Capabilities](../capabilities.md) for the full matrix, the coherence +rules, and the using-clause ordering footgun that page documents +for hand-written gate signatures. + +**Source:** Hansen — *We Need More Optics* ("We can eliminate the +dependence on the BalanceSheet type by requiring a Traversal +instead"; "you are able to specify if you need the guarantees of an +Iso, or the freedom of a Getter"). + +## Serdes free pipes + +**Why:** a Kafka consumer gets an `Array[Byte]` payload, needs to +change one field, and re-emits binary on the producer side. The +conventional answer couples the consumer to the *whole* schema — a +decode into the full case-class tree per message, on a hot path +where that is pure overhead. `AvroPrism` decouples it: the module +depends on one field's optic, not on `OrderEvent`. It IS a +byte-carried optic (`Optic[Array[Byte], Array[Byte], A, A, +Affine]`): it locates the focused field's byte span under a cached +reader schema, decodes only that slice, and writes by splicing the +re-encoded focus back into the payload — no `IndexedRecord` +materialised on either side: + +```scala mdoc:silent +import dev.constructive.eo.avro as eoavro +import dev.constructive.eo.avro.AvroCodec +import java.io.ByteArrayOutputStream +import org.apache.avro.generic.{GenericDatumWriter, GenericRecord} +import org.apache.avro.io.EncoderFactory + +// Hosted in dev.constructive.eo.docs with kindlings-derived +// AvroEncoder / AvroDecoder / AvroSchemaFor givens: +// +// case class OrderEvent(orderId: String, customer: String, total: Double) +import dev.constructive.eo.docs.OrderEvent + +// Stand-in for an inbound Kafka record: serialise an OrderEvent to +// binary under the same schema the prism caches. +val outSchema = summon[AvroCodec[OrderEvent]].schema +val sample = OrderEvent("ord-42", "alice", 99.99) +val sampleBytes: Array[Byte] = + val rec = summon[AvroCodec[OrderEvent]].encode(sample).asInstanceOf[GenericRecord] + val out = new ByteArrayOutputStream() + val encoder = EncoderFactory.get().binaryEncoder(out, null) + val writer = new GenericDatumWriter[GenericRecord](outSchema) + writer.write(rec, encoder) + encoder.flush() + out.toByteArray + +// The optic: walk into `customer` once at construction time, +// reuse on every inbound record. Qualified through the `eoavro` +// alias: eo-circe has a `codecPrism` of its own. +val upperCustomer = eoavro.codecPrism[OrderEvent].customer +``` + +```scala mdoc +// Kafka hot path: bytes in → splice in place → bytes out. No +// re-serialisation step — the prism's write side IS the emit path. +val outBytes: Array[Byte] = + upperCustomer.modify(_.toUpperCase)(sampleBytes) + +// Round-trip witness — decode the output to confirm the customer +// field changed and the rest is preserved. +(upperCustomer.getOption(outBytes), + eoavro.codecPrism[OrderEvent].getOption(outBytes)) +``` + +`.modify` on the byte optic is silent-pass-through — bad bytes, +missing fields, or decode mismatches leave the input payload +untouched (an `Affine` Miss) rather than allocating an `Ior` +chain. That matches the Kafka consumer budget: at-least-once +delivery already implies the consumer must be tolerant of +malformed payloads at the offset commit boundary, and the +per-record allocation cost of `Ior` is a tax on the happy path. +When you DO want the diagnostic — for a dead-letter queue, say — +flip to the record-carried face: +`upperCustomer.record.modify(...)` accepts the same bytes and +returns `Ior[Chain[AvroFailure], IndexedRecord]`; route on the +`Ior.Both` / `Ior.Left` shape from there. + +The cached reader schema is the load-bearing piece: a single +`codecPrism[OrderEvent]` value pins the schema once, and the +parser reuses it across millions of inbound records. For the +schema-registry case where the reader schema arrives at +runtime, use the explicit-schema overload — +`AvroPrism.codecPrism[OrderEvent](runtimeSchema)` — to bypass +the kindlings-derived schema entirely. + +**Source:** cats-eo internal (`AvroPrism`'s byte-carried +default surface). Background framing on the streaming / +Kafka use case lives in the +[Avro integration intro](../integrations/avro.md#why-this-exists). + +## Inspectable cross-format bridges + +**Why:** the consumer reads Avro off a topic and must emit JSON +to a partner API — or receives JSON and must re-emit Avro. Two +modules, two wire formats, and the only thing they genuinely share +is one field. The conventional bridge couples them anyway: decode +the whole record into a case class, build a whole output value, +encode it all. With a byte optic on each side of the seam, only the +moved branch is ever decoded and re-encoded: the full object is +**never constructed** on either format. (`AvroJsonBridgeSpec` +proves this with a counting root codec — both bridge directions run +with zero root decodes and zero root encodes; `AvroJsonBridgeBench` +puts B/op numbers on it.) + +```scala mdoc:silent +import com.github.plokhotnyuk.jsoniter_scala.core.JsonValueCodec +import com.github.plokhotnyuk.jsoniter_scala.macros.JsonCodecMaker +import dev.constructive.eo.jsoniter.JsoniterPrism + +given JsonValueCodec[String] = JsonCodecMaker.make + +// One byte optic per format, same focus type. Drilled once, +// reused for every message. (`JsoniterPrism.fromPath` returns an +// `Either` — a path is data; the literal below is unwrapped here.) +val customerAvro = eoavro.codecPrism[OrderEvent].customer +val customerJson = + JsoniterPrism + .fromPath[String]("$.customer") + .fold(msg => throw new IllegalArgumentException(msg), identity) + +// The JSON side's output skeleton. Placeholders must be VALID +// encodings of the branch type — the splice write decodes the +// current focus before replacing it. +val receiptTemplate: Array[Byte] = + """{"kind":"receipt","customer":"","status":"ok"}""".getBytes("UTF-8") +``` + +```scala mdoc +// Avro → JSON: slice the branch off the Avro wire form, splice +// it into the JSON template. No OrderEvent, no receipt object. +val receipt = customerAvro.getOption(sampleBytes) match + case Some(c) => customerJson.replace(c)(receiptTemplate) + case None => receiptTemplate +new String(receipt, "UTF-8") +``` + +```scala mdoc +// JSON → Avro: read the branch from JSON bytes, splice it into +// existing Avro wire bytes. Same guarantee, reversed. +val inbound: Array[Byte] = + """{"kind":"receipt","customer":"carol","status":"ok"}""".getBytes("UTF-8") +val updatedAvro = customerJson.getOption(inbound) match + case Some(c) => customerAvro.replace(c)(sampleBytes) + case None => sampleBytes + +// Witness (this read DOES construct the object — the bridge +// above never did): the branch moved, the siblings survived. +eoavro.codecPrism[OrderEvent].getOption(updatedAvro) +``` + +Cost model worth knowing: each `.replace` locates the span, +decodes the *current* focus (the `Affine` write carries it), +re-encodes the new one, and allocates one output buffer. Moving +one branch this way is far cheaper than materialising a wide +object — but for many branches per message, or fragment moves +between two *Avro* payloads, prefer `sliceBytes` / `graftBytes` +(no decode at all). + +**Source:** `AvroJsonBridgeSpec` (avro module, `eo.avro.jsoniter`) and +`AvroJsonBridgeBench` (benchmarks) in cats-eo. + +## Re-usable laws for testing + +A decoupled boundary needs a tested contract. Composing optics +means the combinators are already tested — what's left to check is +*your* instances, the optic values your module publishes. The +discipline suites that gate cats-eo's own carriers ship in +`cats-eo-laws`, so a composed optic like `meetingTimes` gets the +full Traversal rule-set in one `checkAll` instead of hand-written +round-trip tests: + +```scala +libraryDependencies += "dev.constructive" %% "cats-eo-laws" % "@VERSION@" % Test +``` + +```scala +import cats.Functor +import dev.constructive.eo.laws.TraversalLaws +import dev.constructive.eo.laws.discipline.TraversalTests + +// meetingTimes walks a plain Meeting, so T is the constant type +// lambda — its Functor is one line. Arbitrary[Meeting] and +// Cogen[Instant] come from your ScalaCheck generators. +given Functor[[X] =>> Meeting] with + def map[A, B](fa: Meeting)(f: A => B): Meeting = fa + +checkAll( + "meetingTimes", + new TraversalTests[[X] =>> Meeting, Instant]: + val laws = new TraversalLaws[[X] =>> Meeting, Instant]: + val traversal = meetingTimes + .traversal, +) +``` + +Every optic family has a matching `FooLaws` / `FooTests` pair — the +[migration guide](../migration-from-monocle.md#what-stays-the-same) has +the import-swap table. + +**Source:** Hansen — *We Need More Optics*, The Startup, + ("Re-usable +tests"). diff --git a/site/docs/cookbook/directory.conf b/site/docs/cookbook/directory.conf new file mode 100644 index 00000000..6061dca0 --- /dev/null +++ b/site/docs/cookbook/directory.conf @@ -0,0 +1,8 @@ +laika.title = Cookbook +laika.navigationOrder = [ + grpc-filter.md + uplc-jvm.md + navigating-structures.md + decoupling.md + effect-threading.md +] diff --git a/site/docs/cookbook/effect-threading.md b/site/docs/cookbook/effect-threading.md new file mode 100644 index 00000000..1e68693d --- /dev/null +++ b/site/docs/cookbook/effect-threading.md @@ -0,0 +1,227 @@ +# Effect threading + +Optics are great at turning a host language's control flow into +just type definitions. The optic fixes *where* — which foci, at +what depth; the effect type `G` you thread through `.modifyF` / +`.modifyA` decides *what the control flow is*: abort on first +failure, accumulate every failure, log alongside the rewrite, defer +into `IO`. Swapping behaviours means swapping `G`, never touching +the optic. + +```scala mdoc:silent +import dev.constructive.eo.optics.{Lens, Optic, Prism, Traversal} +import dev.constructive.eo.optics.Optic.* +import dev.constructive.eo.data.MultiFocus.given // Functor / Foldable / Traverse for MultiFocus[PSVec] (post-fold) +``` + +## Validate-in-place with `modifyF` + +**Why:** the update can fail. Bump `age` by 1, but reject a +negative input with `None` — and keep the validation and the write +as one expression instead of a get / check / set dance. `.modifyF[G]` +lifts an `A => G[B]` through any carrier that admits `Functor[G]`: + +```scala mdoc:silent +import dev.constructive.eo.generics.lens + +case class Visitor(name: String, age: Int) + +val visitorAgeL = lens[Visitor](_.age) +``` + +```scala mdoc +import cats.syntax.functor.* +import cats.instances.option.* + +visitorAgeL.modifyF[Option](age => + if age >= 0 then Some(age + 1) else None +)(Visitor("Alice", 30)) + +visitorAgeL.modifyF[Option](age => + if age >= 0 then Some(age + 1) else None +)(Visitor("Alice", -1)) +``` + +`.modifyA[G]` is the `Applicative[G]` variant — reach for it +when the chain has branching effects to combine (traversal + +validation, for instance); the next recipe is exactly that. +`Witherable`-style filter-and-drop traversal is cross-referenced from +[Penner — *Composable filters using Witherable optics*](https://chrispenner.ca/posts/witherable-optics); +the carrier is deferred to a follow-up release. + +**Source:** Monocle / Haskell `lens` classic `traverseOf`, +generalised. + +## Structure orthogonal to effects + +**Why:** the article's `systemVerifier` insight — "what to do is +decided by the effect type" — in its smallest honest form. Take the +`lineAmounts` traversal from +[Depend only on what's needed](decoupling.md#depend-only-on-what-s-needed) +and thread three different `G`s through the *same* `.modifyA`. The +optic never changes; the control flow does: + +```scala mdoc:silent +// The invoice and its traversal, as in Depend only on what's needed. +case class Line(desc: String, amount: Double) +case class Invoice(fee: Double, lines: List[Line]) + +val lineAmounts = + lens[Invoice](_.lines) + .andThen(Traversal.each[List, Line]) + .andThen(lens[Line](_.amount)) + +val inv = Invoice(5.0, List(Line("widgets", 10.0), Line("gadgets", 20.0))) +``` + +```scala mdoc:silent +import cats.data.{ValidatedNel, Writer} +import cats.syntax.validated.* + +type Checked[A] = ValidatedNel[String, A] +type Audit[A] = Writer[List[String], A] + +val badInv = Invoice(5.0, List(Line("widgets", -10.0), Line("gadgets", -20.0))) +``` + +```scala mdoc +// G = Option: abort on the first bad focus — short-circuit control flow. +lineAmounts.modifyA[Option](a => + if a > 0 then Some(a * 1.1) else None +)(badInv) + +// G = ValidatedNel: visit EVERY focus, accumulate every failure — +// same optic, same lambda shape, completely different control flow. +lineAmounts.modifyA[Checked](a => + if a > 0 then (a * 1.1).validNel else s"non-positive amount: $a".invalidNel +)(badInv) + +// G = Writer: rewrite AND emit an audit line per focus — the effect +// threads a log through the walk without touching the domain types. +lineAmounts.modifyA[Audit](a => + Writer(List(s"scaled $a"), a * 1.1) +)(inv).run +``` + +The same slot takes `IO` (each focus becomes an effect to +sequence), `Eval` (defer the whole rewrite), or an fs2 `Stream` — +the article's original `systemVerifier` pipes every focus of a +traversal through a verification stream with `.modifyF`, and its +shape is exactly the three calls above with a fancier `G`. The +domain types never learn any of this is happening. + +**Source:** Hansen — *We Need More Optics*, The Startup, + ("What +to do is decided by the effect type"). + +## Batch-load nested IDs + +**Why:** every node in a structure carries an ID you need to +resolve against a database, and the naive `.modifyA` fires one +query per node — the classic N+1. Use the same traversal twice: +`foldMap` collects every ID in one pass, you issue a single batched +query, then `.modify` distributes the results back. 100×–300× +fewer queries with no change to the domain types: + +```scala mdoc:silent +case class Node(id: Int, label: String) +case class Payload(id: Int, body: String) + +// Stand-in for a DB-backed batch fetch. +def fetchAll(ids: List[Int]): Map[Int, Payload] = + ids.map(i => i -> Payload(i, s"body-$i")).toMap + +val eachLeaf = Traversal.each[List, Node] +``` + +```scala mdoc +val nodes = List(Node(1, "a"), Node(2, "b"), Node(3, "c")) + +// Pass 1: collect every ID into a single list via foldMap. +val allIds = eachLeaf.foldMap((n: Node) => List(n.id))(nodes) + +// Pass 2: issue ONE query for the whole set. +val byId = fetchAll(allIds) + +// Pass 3: broadcast the resolved payloads back through the same +// traversal. The structure is preserved; only the labels shift. +eachLeaf.modify(n => n.copy(label = byId(n.id).body))(nodes) +``` + +The pattern generalises to trees, graphs, nested containers, +anything with a `Traverse`. The two-pass idiom is what cats-eo's +`foldMap` + `modify` pair already enables out of the box — no +cats-eo-specific API to learn. + +**Source:** Penner — *Using traversals to batch database +queries*, . + +## Persist-and-stamp decode free + +**Why:** a `PUT` (or `POST`) handler receives a draft entity on the +wire, persists it, and must return the *same* entity enriched with +the database-assigned id. The id only exists after the effectful +store runs, so this is the classic "set one field to the result of +an effect" shape — and a derived id-lens makes the stamp a one-liner. +With circe carrying the value across the wire on both ends, the +whole handler is a short pipeline: `decode → store → stamp → +encode`. + +```scala mdoc:silent +import io.circe.Json +import io.circe.syntax.* +import io.circe.parser.decode +import dev.constructive.eo.generics.lens + +// cats.Eval stands in for your effect type (cats-effect IO, ZIO, +// Future…) — deferred, has map/flatMap, already on the classpath. +import cats.Eval + +// Hosted in dev.constructive.eo.docs with a kindlings-derived +// Codec.AsObject given: +// +// final case class BalanceSheet(id: Long, owner: String, total: Double) +import dev.constructive.eo.docs.BalanceSheet + +// One derived lens onto the id — no hand-written getter/setter, and +// it works even though `id` shares the case class with two other fields. +val sheetId = lens[BalanceSheet](_.id) + +// The effectful store: inserts the row, hands back the generated id. +def save(sheet: BalanceSheet): Eval[Long] = Eval.always { + val _ = sheet // pretend: INSERT … RETURNING id + 42L +} + +// Persist, then stamp the returned id back onto the object with the +// lens — `save(...).map(sheetId.replace(_)(sheet))`. +def store(sheet: BalanceSheet): Eval[BalanceSheet] = + save(sheet).map(sheetId.replace(_)(sheet)) + +// The whole PUT handler: request bytes in, response bytes out. +def handlePut(body: String): Eval[String] = + decode[BalanceSheet](body) match + case Right(draft) => store(draft).map(_.asJson.noSpaces) + case Left(err) => Eval.now(Json.obj("error" -> err.getMessage.asJson).noSpaces) +``` + +```scala mdoc +// Incoming PUT body — `id` is a placeholder the store overwrites. +val request = """{"id":0,"owner":"Acme Corp","total":1234.5}""" + +handlePut(request).value +``` + +Every step earns its place: `decode` is the wire → domain hop +(circe), `store` is your effect, the stamp is the derived lens +standing in for a hand-written `sheet.copy(id = newId)`, and +`asJson` is the domain → wire hop back out. The lens is the only +optic, and it stays a reusable value — compose it further +(`orderLens.andThen(sheetId)`) the moment the entity nests inside a +larger request. In production you'd reach for an opaque `DbId` +rather than a bare `Long`, and likely a separate `Draft` type +without the id field; the shape of the pipeline doesn't change. + +**Source:** the lens-stamps-the-effect-result pattern, +; +see also [Generics → Composing into pipelines](../generics.md#composing-derived-optics-into-pipelines). diff --git a/site/docs/cookbook/grpc-filter.md b/site/docs/cookbook/grpc-filter.md new file mode 100644 index 00000000..5c9a3a35 --- /dev/null +++ b/site/docs/cookbook/grpc-filter.md @@ -0,0 +1,346 @@ +# Allocation-free stream triage demo + +> **A demo, not a module.** The code lives in +> [`demos/grpc-filter`](https://github.com/Constructive-Programming/eo/tree/main/demos/grpc-filter) +> and is not published. It shows what [eo-jsoniter](../integrations/jsoniter.md)'s +> byte-level optics buy a streaming service: a filter on a gRPC stream that +> classifies each message and forwards only the ones a policy keeps, without +> ever decoding a message, written with [fs2](https://fs2.io). + +## The idea: read three values, not 64 fields + +An upstream service streams issues to a downstream one. They travel as gRPC +messages with JSON payloads, raw bytes off a TCP socket. Each issue has 64 +fields. One of them, `detail`, says what kind of issue it is: a bug, an +incident or a feature request, each with its own fields, nested up to three +objects deep. A filter between the two services keeps: + +- incidents of severity 1 or 2 on a critical service, +- bugs whose top stack frame is in a watched module, +- requests with enough votes from customers on the plans that matter, + +and drops everything else. + +To decide, the filter needs the kind and one or two values of the +kind's own. The usual way decodes the whole issue to read them: 64 fields, +lists, nested objects, about 8.5 KB of objects per 4 KB message. This filter +reads them where they are, in the bytes, with eo-jsoniter's prisms. It never +builds an `Issue`, and it forwards the messages it keeps exactly as they came. + +``` +upstream service ──TCP──▶ filter ──TCP──▶ downstream service + │ + bytes ─ GrpcFrames.splitFrames ─▶ message ─ Wire.detail ⇀ DetailJson ─ Policy.keep ─▶ keep? + │ + bytes ◀─ GrpcFrames.joinFrames ◀─ the kept messages, byte for byte ◀────────────────────┘ +``` + +## The stream + +gRPC frames every message the same way, whatever its encoding: one byte saying +whether the message is compressed, its length as a 4-byte big-endian unsigned +integer, then the message (the *Length-Prefixed-Message* of the +[gRPC over HTTP/2 spec](https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md)). +gRPC is not tied to protobuf: a service can carry JSON (`application/grpc+json`), +and this one does. + +`GrpcFrames` holds the framing as two fs2 pipes over raw bytes. `splitFrames` cuts +messages out of the bytes in whatever pieces the socket reads them, and fails +the stream on a compressed frame (no message encoding is negotiated here), on a +message over gRPC's default 4 MiB limit, or on bytes that end inside a frame. +`joinFrames` frames messages again, without copying them. + +```scala +socketBytes // Stream[F, Byte], as the socket reads it + .through(GrpcFrames.splitFrames()) // Stream[F, Array[Byte]]: one per message + .through(GrpcFrames.joinFrames) // Stream[F, Byte]: the same frames again +``` + +The demo puts these frames straight on TCP rather than inside HTTP/2 streams: +the subject is what happens to the message bytes, and over HTTP/2 the same +frames arrive as the body of a call. So the demo does not speak HTTP/2, and has +no headers, trailers or status codes. + +## The message + +```scala +final case class Issue( + id: Long, + key: String, + title: String, + description: String, + reporter: Person, // Person → Team → Org: three objects deep + // … 55 more … + detail: Detail, // the 61st field + tags: Map[String, String], + checklist: List[CheckItem], + revision: Long, +) + +enum Detail: + case Bug(component: String, build: String, reproducible: Boolean, trace: Trace) + case Incident(severity: Severity, startedAt: Long, mitigated: Boolean, impact: Impact) + case Request(product: String, votes: Votes, quarter: String, requester: Customer) + +final case class Impact(users: Long, errorRate: Double, service: Service) +final case class Service(name: String, tier: Tier, region: String) // detail → impact → service +``` + +jsoniter-scala writes a `Detail` as an object whose first field, `"type"`, names +its case: `{"type":"Incident","severity":"Sev1",…}`. + +The issues come from ScalaCheck generators that +[scalacheck-derived](https://github.com/martinhh/scalacheck-derived) derives +from the model, one per case class and enum, from its fields and cases: + +```scala +object Issues: + given Arbitrary[String] = /* words; one in twenty is something JSON must escape, or non-ASCII */ + given Arbitrary[Double] = Arbitrary(Gen.choose(0.0, 1e6)) // JSON has no NaN or infinities + given Arbitrary[Module] = Arbitrary(Gen.oneOf(modules)) + given Arbitrary[Votes] = Arbitrary(Gen.choose(0, 200).map(Votes(_))) + given Arbitrary[Detail] = deriveArbitrary + given Arbitrary[Issue] = deriveArbitrary +``` + +Only the leaves are written by hand, so a field added to the model is generated +without touching the generators. A third of the issues are of each kind. The +stream draws them at size 7: up to seven words in a text and seven elements in +a list, which makes a message 2.6 to 5.9 KB, 4.1 KB on average. The property +tests draw them at every size from 2 to 40. + +## Reading without decoding + +`Wire` holds the optics over a message's bytes. Each scans the bytes for its +path and decodes only what it focuses: + +```scala +object Wire: + opaque type DetailJson = Array[Byte] // a detail's own JSON, sliced out of a message + + // Message ⇀ its detail's bytes: one scan, past the 60 fields in front; nothing decoded. + val detail: Focus[Array[Byte], DetailJson] = JsoniterPrism[Issue].field(_.detail).raw + + given kind: Focus[DetailJson, Kind] = /* JsoniterPrism.fromPath[Kind]("$.type") */ + given severity: Focus[DetailJson, Severity] = JsoniterPrism[Detail.Incident].field(_.severity) + given tier: Focus[DetailJson, Tier] = + JsoniterPrism[Detail.Incident].field(_.impact).field(_.service).field(_.tier) + given module: Focus[DetailJson, Module] = + JsoniterPrism[Detail.Bug].field(_.trace).field(_.frames).at(0).field(_.module) + given plan: Focus[DetailJson, Plan] = + JsoniterPrism[Detail.Request].field(_.requester).field(_.contract).field(_.plan) + given votes: Focus[DetailJson, Votes] = JsoniterPrism[Detail.Request].field(_.votes) +``` + +Reading takes two steps. `detail` scans the message once and hands out the +detail's own bytes, a few hundred of them (`.raw` focuses the encoded slice +rather than a decoded value). The other optics read the policy's leaves from +that slice: a kind, an enum, a number. `DetailJson` is an opaque type, so the +detail's optics can't be pointed at a whole message by mistake. + +The paths are typed: `.field(_.impact)` is checked against the model when +`Wire` compiles, so renaming a field breaks the build, not the filter. Each +case's paths start from that case's own root, `JsoniterPrism[Detail.Incident]`, +because the detail's JSON is that case's JSON plus its `"type"`. Only the +kind's path is a string: `"type"` is the codec's tag, not a field of the model. + +Leaf optics of different cases don't check the kind: `tier` on a bug's detail +misses only because a bug has no `impact`. The policy classifies before it +reads a leaf, and the tests check that the bytes and the decoded issue are +always judged alike. + +## One policy, two readers + +The policy is written once, against capabilities, and doesn't know what it +reads: + +```scala +final case class Policy(urgent: Set[Severity], tiers: Set[Tier], modules: Set[Module], + plans: Set[Plan], minVotes: Votes): + + def keep[D](detail: D)(using + kind: CanGetOption[D, Kind], + severity: CanGetOption[D, Severity], + tier: CanGetOption[D, Tier], + module: CanGetOption[D, Module], + plan: CanGetOption[D, Plan], + votes: CanGetOption[D, Votes], + ): Boolean = + kind.getOption(detail) match + case Some(Kind.Incident) => + severity.getOption(detail).exists(urgent) && tier.getOption(detail).exists(tiers) + case Some(Kind.Bug) => module.getOption(detail).exists(modules) + case Some(Kind.Request) => + plan.getOption(detail).exists(plans) && votes.getOption(detail).exists(_.count >= minVotes.count) + case None => false +``` + +With `Wire`'s givens in scope, `D` is `DetailJson` and the policy reads bytes. +With `Model`'s, `D` is a decoded `Detail`, read through eo-generics' derived +prisms and lenses: + +```scala +object Model: + given tier: Focus[Detail, Tier] = + prism[Detail, Detail.Incident] + .andThen(lens[Detail.Incident](_.impact)) + .andThen(lens[Impact](_.service)) + .andThen(lens[Service](_.tier)) + // … one per leaf, and the kind +``` + +The baseline uses `Model`, after decoding the whole issue. So do the tests, +which check that both readers judge every message alike. That is what +*consume via capability, construct via optic* buys: the code that decides +never names the optic or the format it reads. + +```scala +def optics(policy: Policy)(message: Array[Byte]): Verdict = + import Wire.given + Wire.detail.getOption(message) match + case Some(detail) => Verdict(Wire.kind.getOption(detail), policy.keep(detail)) + case None => Verdict(None, keep = false) + +def decoding(policy: Policy)(message: Array[Byte]): Verdict = + import Model.given + Issue.decode(message).map(_.detail) match + case Some(detail) => Verdict(Some(Kind.of(detail)), policy.keep(detail)) + case None => Verdict(None, keep = false) +``` + +A message of a kind the demo doesn't know, from a newer producer say, has no +kind: the policy drops it, and the tally counts it as unclassified. The optics +read it like any other message. A decoder of the full model rejects the whole +message, and the baseline drops it for that reason. + +## The filter + +```scala +def filter[F[_]: Async](verdict: Array[Byte] => Verdict, tally: Ref[F, Tally]): Pipe[F, Byte, Byte] = + _.through(GrpcFrames.splitFrames()) + .chunks + .evalMap { messages => + Async[F].delay(messages.map(message => (message, verdict(message)))).timed.flatMap { + (took, judged) => + tally.update(_.add(judged, took)).as(judged.collect { case (m, Verdict(_, true)) => m }) + } + } + .unchunks + .through(GrpcFrames.joinFrames) +``` + +`Net` puts it between two sockets. The upstream service streams frames to +whoever connects. The filter reads up to 256 KiB per read (fs2's `reads` asks +for 8 KiB, and on loopback the reads were the bottleneck), triages, and writes +what it keeps to the downstream service. Reading, judging and writing run on +separate fibers, with a few chunks buffered between them (`prefetchN`). The +downstream service decodes everything it receives and checks it against the +policy. + +`sbt "demoGrpcFilter/run 200000"` runs the three of them on the loopback +interface, once with each filter (after an untimed run of each to warm the JIT +compiler up): + +``` +eo-jsoniter optics: 200000 messages, 813.3 MB in 2.21 s: + 91 k messages/s, 368 MB/s + judging them took 7.1 µs per message; the rest is sockets and framing + kind seen kept + Bug 66199 8205 + Incident 63289 10745 + Request 70512 12500 + none 0 0 + downstream received 31450 messages (31450 kept); 31450 decode as issues, and the policy keeps 31450 of them +full decode (baseline): 200000 messages, 813.3 MB in 4.16 s: + 48 k messages/s, 196 MB/s + judging them took 16.8 µs per message; the rest is sockets and framing + … +``` + +## Correctness + +`sbt demoGrpcFilter/test`: + +| suite | checks | +|---|---| +| `GrpcFramesSpec` | any messages, framed and cut into arbitrary pieces, decode back whole; the frame layout; the size limit (unsigned lengths included); compressed frames and truncated bytes fail the stream | +| `TriageSpec` | the optics judge every message exactly as decoding does: 3 000 issues the size the stream's are, under the default policy and under 100 random ones, and 500 random issues of every size under random policies, every kind both kept and dropped; the slice `Wire.detail` hands out is the detail's own JSON; pretty-printed JSON is read the same; an unknown kind, a missing `detail` and bytes that aren't JSON are dropped as unclassified | +| `TriageSpec` (laws) | `Model`'s optics pass cats-eo-laws' `OptionalTests` | +| `PipelineSpec` | the whole filter, in memory with the bytes cut at random, and over real sockets: it forwards exactly the messages the decoded issues say to keep, byte for byte and in order, and its tally adds up; a malformed frame fails it | + +## Benchmarks + +JMH, in +[`demos/grpc-filter-bench`](https://github.com/Constructive-Programming/eo/tree/main/demos/grpc-filter-bench). +Average time per message in **ns (lower is better)**, error = 99.9 % +confidence half-width; allocation per message from `-prof gc` +(`gc.alloc.rate.norm`). Measured on a 4-vCPU Intel Xeon (2.1 GHz) container, +OpenJDK 21.0.12, fs2 3.14.0, jsoniter-scala 2.41.2; three forks, 5 +measurements after 3 warm-ups. A shared container is not a quiet benchmark +machine: read the ratios, not the absolute values. + +### One message + +`TriageBench` judges a pool of 256 messages, small enough to stay in cache, as +a message just read from a socket is. `decode` decodes the whole issue and runs +the policy on its `Detail`; `optics` slices out the detail and runs the policy +on its bytes; `detail` is the slicing alone. + +`layout` is where `detail` sits in the message: `late` is where the codec +writes it (the 61st of 64 fields, about 90 % of the way in), `first` is the +same message with `detail` moved to the front. + +| layout | `decode` | `optics` | `detail` | speed-up | allocation | +|---|--:|--:|--:|--:|--:| +| late | 13 833 ± 620 ns, 8 557 B | 5 929 ± 450 ns, 705 B | 5 371 ± 219 ns, 367 B | 2.3× | 12× less | +| first | 13 403 ± 707 ns, 8 548 B | 1 207 ± 78 ns, 664 B | 711 ± 63 ns, 431 B | 11× | 13× less | + +Decoding reads every byte, wherever `detail` is. The scan reads the bytes in +front of `detail`, so its cost depends on where the producer puts the field: +in the codec's own layout, nine tenths of the optics' time is the scan past +the 60 fields in front of `detail`. It skips them at about the speed of +jsoniter-scala's own `skip`. What it saves either way is the building: 700 +bytes allocated per message instead of 8.6 KB. + +### The whole filter + +`StreamBench` runs the filter's pipe in memory: 1 024 messages' frames, +arriving in 256 KiB pieces, are cut into messages, judged, tallied and the kept +ones framed again. + +| judge | per message | allocation | +|---|--:|--:| +| `decode` | 17 695 ± 845 ns | 13 917 B | +| `optics` | 8 911 ± 449 ns | 6 001 B | + +2.0× faster, with 2.3× less allocation. The pipe adds 3 to 4 µs and 5.3 KB per +message either way. Most of that is copying each message out of the +socket's chunks into an array of its own (4 KB), because a `JsoniterPrism` +reads a whole `Array[Byte]`. + +## Limits + +- **No HTTP/2.** The frames travel on plain TCP: no headers, trailers or + status codes, and no compression (`grpc-encoding` is never negotiated, so a + compressed frame fails the stream). +- **The scan's cost follows the layout.** A field late in a large message + costs most of a read through the message. An optic only saves the reading + past the field and the building of what it doesn't focus. +- **Copies.** Each message is copied out of the socket's chunks, and `.raw` + copies the detail's bytes, since eo-jsoniter's optics read a whole array. +- **Leaf optics trust the classification.** They don't check the kind + themselves; the policy classifies first. + +## Running it + +```sh +# the suite +sbt demoGrpcFilter/test +# the three services on loopback, with each filter +sbt "demoGrpcFilter/run 200000" +# the benchmarks +sbt "demoGrpcFilterBench/Jmh/run -prof gc" +``` + +The demo and its benchmarks sit outside the root aggregate; CI runs the suite +and compiles the benchmarks as their own step. diff --git a/site/docs/cookbook/navigating-structures.md b/site/docs/cookbook/navigating-structures.md new file mode 100644 index 00000000..eb3ae19f --- /dev/null +++ b/site/docs/cookbook/navigating-structures.md @@ -0,0 +1,425 @@ +# Navigating structures + +One branch of a sum; every node of a recursive tree; a sparse walk +over a collection; several fields at once; and the same moves over +JSON you never decode. The recipes ramp by *shape*, and the point +throughout is that no step dead-ends: whatever you just focused, +the next `.andThen` keeps going. + +```scala mdoc:silent +import dev.constructive.eo.optics.{Lens, Optic, Prism, Traversal} +import dev.constructive.eo.optics.Optic.* +import dev.constructive.eo.data.MultiFocus.given // Functor / Foldable / Traverse for MultiFocus[PSVec] (post-fold) +``` + +## Visit contingent fields + +**Why:** you want to rewrite one variant of a sum type — the bound +variable of a `Var` node in an AST, say — and leave every other +case untouched, without a hand-written `match` that re-builds the +misses. Both hops derive from the type itself: `prism[S, A]` +matches the branch, `lens[S](_.field)` reaches inside it, and one +`.andThen` chains them — no pattern match, no `copy` plumbing, no +optic written by hand. + +```scala mdoc:silent +// The AST under edit — hosted at package level in +// dev.constructive.eo.docs (macro-derived optics need top-level +// targets; mdoc wraps every fence in an object): +// +// enum Expr: +// case Var(name: String) +// case App(f: Expr, x: Expr) +// case Lam(bind: String, body: Expr) +import dev.constructive.eo.docs.Expr +import dev.constructive.eo.generics.{lens, prism} + +val varName = prism[Expr, Expr.Var].andThen(lens[Expr.Var](_.name)) + +// The focus arrives as the case's `(name: String)` NamedTuple — +// the selector covers Var's only field, so the derived optic is a +// full BijectionIso rather than a partial Lens. +val upperVar = varName.modify(v => (name = v.name.toUpperCase)) +``` + +```scala mdoc +// Hit branch: the Var's name is uppercased. +upperVar(Expr.Var("x")) + +// Miss branch: passes through, the Lam is untouched. +upperVar(Expr.Lam("y", Expr.Var("y"))) +``` + +The derived Prism rides the `Either` carrier and the derived field +optic composes across the seam through the same `.andThen` as +everything else on this page. When you need a Prism the macro can't +express — a validating match, a normalising re-build — the +hand-rolled `Prism[S, A](sel, emb)` constructor takes the match +explicitly; see [Generics → `prism[S, A]`](../generics.md) for what +the derivation covers (enums, sealed traits, union types) and the +full story on the NamedTuple focus. + +**Source:** Baeldung — *Monocle Optics*, +; framing from +Wlaschin — *Domain Modeling Made Functional*, ch. 4, +. + +## Visit across whole trees + +`varName` edits exactly the node you hand it: `Var("x")` hits, +`Lam(...)` misses — the nested `Var("y")` inside stays untouched, +because a Prism looks at the *top* of the value, not into it. One +more derivation closes the gap: `plate[Expr]` derives the `Plated` +instance (which fields recurse, which are leaves), and +`everywhere` turns it into a recursive optic that reaches every +sub-term. The **same `varName` optic** then composes into it: + +```scala mdoc:silent +import dev.constructive.eo.optics.Plated +import dev.constructive.eo.generics.plate + +given Plated[Expr] = plate[Expr] + +// `everywhere` is `transform` in optic form: everywhere.andThen(d).modify(g) +// applies `d` at every node, bottom-up. Reuse `varName` from above. +val everyVarName = Plated.everywhere[Expr].andThen(varName) + +val upperEveryVar = everyVarName.modify(v => (name = v.name.toUpperCase)) +``` + +```scala mdoc +val term = Expr.App(Expr.Var("f"), Expr.Lam("y", Expr.Var("y"))) + +// varName alone: the top of `term` is an App, not a Var — a miss, +// nothing changes. +upperVar(term) + +// everywhere ∘ varName: every variable at every depth — f -> F and +// the nested y -> Y. The Lam binder "y" (a String, not a Var node) +// is left alone. +upperEveryVar(term) +``` + +That pair of calls is the whole comparison: `varName` is the +surgical single-node edit, `everyVarName` the same edit quantified +over the tree — and the step between them is one derivation plus +one `.andThen`, not a rewrite. The `.modify` runs at every node, +bottom-up and stack-safe to any depth (a million-node tree, or a +100k-deep degenerate spine, won't overflow). When the rewrite is easier +as a plain per-node function, `Plated.transform(f)` is the same engine; +`Plated.universe` / `children` are the read side (every sub-term / +immediate children); `rewrite` repeats an `Expr => Option[Expr]` rule +to a fixpoint. The derivation follows the *exact self-type* rule — +only `Expr`-typed fields are recursion points, so `Lam`'s +`bind: String` stays a leaf. See [Generics → `plate[S]`](../generics.md). + +**Source:** Mitchell & Runciman — *Uniform Boilerplate and List +Processing* (Uniplate), +; +Haskell `lens` `Control.Lens.Plated`, +. + +## Visit through arbitrary structure + +**Why:** you have a list of results, some `Ok` and some `Err`, and +want to bump only the successes — leaving the failures exactly +where they are. Walking the container *and* matching the branch in +one pass is the "sparse traversal" that's genuinely annoying to +hand-roll; here it's a one-liner. + +```scala mdoc:silent +import scala.collection.immutable.ArraySeq + +// Result (Ok | Err) is hosted in dev.constructive.eo.docs like Expr +// above — enum macro targets need a package-level home under mdoc. +import dev.constructive.eo.docs.Result + +val okP = prism[Result, Result.Ok] + +val bumpOks = + Traversal.each[ArraySeq, Result] + .andThen(okP) + // Result.Ok is single-field, where the full-cover lens macro returns + // an Iso with a NamedTuple focus — hand-write the Int lens instead. + .andThen(Lens[Result.Ok, Int](_.value, (o, v) => o.copy(value = v))) +``` + +```scala mdoc +bumpOks.modify(_ + 1)( + ArraySeq(Result.Ok(10), Result.Err("nope"), Result.Ok(20)) +) +``` + +Performance: the `PowerSeriesPrismBench` suite measures this +shape at ~5× over a hand-rolled naive loop — and that's the +published worst case for `each`, not the typical one. The +per-benchmark curve lives in +[benchmarks → PowerSeries with Prism inner](../benchmarks.md#powerseries-traversal-with-downstream-composition). + +**Source:** Penner — *Optics By Example* ch. 8 (Traversal +Actions) + ch. 10 (Missing Values), +. + +## Isolate what you need + +**Why:** you have a basket of line items and want one number — +total value — without writing a fold by hand or threading two +fields through a loop. The `lens[S](_.a, _.b)` macro focuses both +`quantity` and `price` as a single Scala 3 `NamedTuple`, so the +multiply-and-sum drops straight into `foldMap`. + +```scala mdoc:silent +import cats.instances.list.given // Traverse[List] for .each +import cats.instances.double.given // Monoid[Double] for .foldMap +import dev.constructive.eo.generics.lens + +case class OrderItem(sku: String, quantity: Int, price: Double) + +// Focus both pricing fields at once, then walk every item. +val lineValue = + Traversal.each[List, OrderItem] + .andThen(lens[OrderItem](_.quantity, _.price)) +``` + +```scala mdoc +val inventory = List( + OrderItem("apple", 3, 9.99), + OrderItem("pear", 10, 2.50), + OrderItem("lobster", 2, 45.00), +) + +// One pass: see each item's (quantity, price) NamedTuple, multiply, +// and let the Double monoid sum the line values into a grand total. +lineValue.foldMap(nt => nt.quantity * nt.price)(inventory) +``` + +The focus arrives in selector order, so `nt.quantity` and +`nt.price` are named — the lambda reads like the business rule it +encodes. The same `lineValue` optic also writes (`.modify` to +re-price every line); the fold is just the read-side escape. See +[Generics → Multi-field Lens](../generics.md) for the full treatment +of the NamedTuple focus. + +**Source:** cats-eo internal; fold framing from Penner — *Optics +By Example* ch. 7, . + +## Compute aggregations + +A `Lens` sees one value; a `Traversal` visits a container's elements. +`Indexed` addresses a fixed index space, while `MultiFocus[F]` +supports container traversal and aggregation. The two recipes below +show that distinction; see the [MultiFocus reference](../multifocus.md). + +### Recipe A — Adjust a fixed-index configuration + +**Why:** an environment-indexed configuration has a path for each +coordinate. Read one, rewrite each with its index visible, or fill +the whole space without enumerating or comparing indexes. + +```scala mdoc:silent +import cats.Representable +import cats.instances.function.given +import java.nio.file.Path +import dev.constructive.eo.optics.Indexed +import dev.constructive.eo.data.MultiFocus +import dev.constructive.eo.data.MultiFocus.given +import dev.constructive.eo.data.MultiFocus.{collectList, collectMap} + +enum Env: + case Dev, Prod + +val readerRepr = summon[Representable.Aux[[a] =>> Env => a, Env]] +val configGlass = Indexed.representable[[a] =>> Env => a, Path](readerRepr) +val config: Env => Path = + case Env.Prod => Path.of("/opt/app/") + case Env.Dev => Path.of("/dev/app/0292103") +``` + +```scala mdoc +configGlass.at(Env.Dev)(config) +val adjustedConfig = configGlass.modify((env, path) => + if env == Env.Dev then path.resolve("logs") else path +)(config) +(adjustedConfig(Env.Dev), adjustedConfig(Env.Prod)) +val replacePath = Path.of("/") +val replacedConfig = configGlass.replace(replacePath)(config) +(replacedConfig(Env.Dev), replacedConfig(Env.Prod)) +``` + +```scala mdoc:silent +require(configGlass.at(Env.Dev)(config) == Path.of("/dev/app/0292103")) +require(configGlass.at(Env.Prod)(config) == Path.of("/opt/app/")) +require(adjustedConfig(Env.Dev) == config(Env.Dev).resolve("logs")) +require(adjustedConfig(Env.Prod) == config(Env.Prod)) +require(replacedConfig(Env.Dev) == replacePath) +require(replacedConfig(Env.Prod) == replacePath) +``` + +This representable constructor returns `Indexed.Grate`, the +`X = Unit` specialization. Other glasses retain residual context. +Nested `andThen` uses product indexes to address the full grid; +Unit-context normalization and generic classical-family seams remain +open. For effectful per-element rewrites, use container traversal. + +For nested environment/channel settings, type-changing writes, and +labelled sensor grids that retain per-row metadata, continue with +[Indexed: Grates and Glasses](../indexed.md). + +**Source:** Penner — *Grate: yet another optic*, +. + +### Recipe B — Summarise a batch, then broadcast or collapse + +**Why:** a column of readings from a batch needs to become a +*report*: each row pairing the original value with its distance +from the batch average, plus the batch-level statistics — mean, +variance, standard deviation — computed once. The same +`MultiFocus[F]` optic covers every shape the report needs: +broadcast a summary into every position, collapse the batch to a +single footer value, rewrite every row *relative to the batch* in +one expression, or fold out the statistical moments in one pass — +you pick the flavour at the call site: + +```scala mdoc:silent +case class ReportRow(value: Double, distanceFromMean: Double) + +val readings = List(12.0, 14.0, 11.0, 15.0) + +val readingsMF = MultiFocus.apply[List, Double] +``` + +```scala mdoc +// (1) Baseline column — .collectMap broadcasts the batch mean back +// into every position, keeping the shape. +readingsMF.collectMap[Double](xs => xs.sum / xs.size)(readings) + +// (2) Report footer — .collectList collapses the batch to a single +// summary value, whatever the input length. +readingsMF.collectList(xs => xs.sum / xs.size)(readings) + +// (3) The report rows, in one expression — .collectWith is the +// algebraic-lens universal: the aggregate sees the whole batch +// ONCE, computes the baseline, and returns the per-row rewrite. +// The pApply factory makes the walk type-changing, so each +// Double becomes a ReportRow pairing the original with its +// distance from the average. +MultiFocus.pApply[List, Double, ReportRow].collectWith { xs => + val mean = xs.sum / xs.size + v => ReportRow(v, v - mean) +}(readings) + +// (4) The batch-level statistics for the report footer: count, sum +// and sum of squares in ONE foldMap pass via the tuple monoid, +// then the moments fall out. +val (n, sum, sumSq) = readingsMF.foldMap(v => (1, v, v * v))(readings) +val mean = sum / n +val variance = sumSq / n - mean * mean // population variance +val stdDev = math.sqrt(variance) +``` + +**Which flavour?** + +- `.collectMap[B](agg: F[A] => B)` keeps the shape — the summary + lands in every position. Reach for it when a downstream step needs + the aggregate *alongside* the originals (share-of-total, a + normalisation baseline). +- `.collectList(agg: List[A] => B)` supplies a singleton focus vector + to reconstruction with the observed context. This identity-backed + `readingsMF` returns a singleton List, but other optics can preserve + surrounding structure or reject a shape/count mismatch. It is not + a general promise of a single-element source. +- `.collectWith(agg: F[A] => A => B)` is the general map-shaped + collect — the aggregate sees the batch once and the returned + function runs per position, so batch-relative rewrites + (share-of-total, distance-from-mean) are one expression. + `collectMap` is its constant-function special case, and the + type-changing `MultiFocus.pApply[F, A, B]` factory un-pins the + element type so the rewrite can emit a different row type + (step 3). Only `collectList` stays separate — its singleton + collapse changes the focus count, which no map-shaped combinator + can express. + +Why distinct flavours, rather than one derived automatically? The +[MultiFocus reference](../multifocus.md#why-two-collect-variants) has +the answer. + +**Sources:** Penner — *Kaleidoscopes: lenses that never die*, +; Penner — +*Algebraic lenses*, . + +## Edit JSON without decoding + +**Why:** you need to change one field deep inside a JSON payload +and pass the rest through untouched — no full decode into a +case-class tree, no re-encode, no decoder for the siblings you +never read. Navigation doesn't stop at the case-class boundary: +`codecPrism[S]` walks circe's `Json` directly, and only the focused +leaf is materialised as `A`: + +```scala mdoc:silent +import dev.constructive.eo.circe.codecPrism +import io.circe.syntax.* + +// The payload's case classes — hosted at package level in +// dev.constructive.eo.docs with kindlings-derived codecs (heavy +// derivation macros run in compiled sources, not mdoc fences): +// +// case class UserAddress(street: String, zip: Int) +// object UserAddress: +// given Codec.AsObject[UserAddress] = KindlingsCodecAsObject.derived +// +// case class SiteUser(name: String, address: UserAddress) // ditto +import dev.constructive.eo.docs.{SiteUser, UserAddress} + +val userStreet = codecPrism[SiteUser].address.street +``` + +```scala mdoc +val userJson = SiteUser("Alice", UserAddress("Main St", 12345)).asJson +userStreet.modifyUnsafe(_.toUpperCase)(userJson).noSpacesSortKeys +``` + +The +[`OrderCirceBench`](https://github.com/Constructive-Programming/eo/blob/main/benchmarks/src/main/scala/dev/constructive/eo/bench/OrderCirceBench.scala) +suite shows this edit is **flat in document size** — ~1.3 µs whether the record +is small or large — while the decode / `.copy` / re-encode path scales with the +whole payload, so the gap grows from ~3× on a tiny record to ~160× on a large +one. circe-optics' analogous `root.user.address.street` surface forces a full +decode per level; cats-eo's cursor walk does not. + +The `*Unsafe` suffix opts out of the failure channel; the default +`modify` returns `Ior[Chain[JsonFailure], Json]`, so a silent +no-op edit tells you which step refused — the +[circe integration's failure flow](../integrations/circe.md#failure-flow) +covers routing it. + +**Source:** cats-eo internal (`JsonPrism`); related to circe-optics' +`root.*` idiom . + +### …and every element of a JSON array + +Walk an array without materialising it as a Scala collection; +only the focused leaf of each element is decoded. The `.each` +step splits the Prism into a `JsonTraversal`: + +```scala mdoc:silent +// Same hosting pattern — kindlings-derived Codec.AsObject on both: +// +// case class Item(name: String, price: Double) +// case class Basket(owner: String, items: Vector[Item]) +import dev.constructive.eo.docs.{Basket, Item} +``` + +```scala mdoc +val basket = Basket("Alice", Vector(Item("apple", 1.0), Item("pear", 2.0))) +val everyItemName = codecPrism[Basket].items.each.name + +everyItemName.modifyUnsafe(_.toUpperCase)(basket.asJson).noSpacesSortKeys +``` + +Per-element failures to decode accumulate into +`Ior.Both(chain, partialJson)` on the default surface — the +[circe integration's failure flow](../integrations/circe.md#failure-flow) +shows how to route the chain. + +**Source:** cats-eo internal (`JsonTraversal`). diff --git a/site/docs/demos/uplc-jvm.md b/site/docs/cookbook/uplc-jvm.md similarity index 99% rename from site/docs/demos/uplc-jvm.md rename to site/docs/cookbook/uplc-jvm.md index dcc97766..a87cc8d8 100644 --- a/site/docs/demos/uplc-jvm.md +++ b/site/docs/cookbook/uplc-jvm.md @@ -1,4 +1,4 @@ -# UPLC → JVM bytecode +# Blockchain smart contract compilation demo > **A demo, not a module.** The code lives in > [`demos/uplc-jvm`](https://github.com/Constructive-Programming/eo/tree/main/demos/uplc-jvm) diff --git a/site/docs/demos/directory.conf b/site/docs/demos/directory.conf deleted file mode 100644 index 7486263b..00000000 --- a/site/docs/demos/directory.conf +++ /dev/null @@ -1,4 +0,0 @@ -laika.title = Demos -laika.navigationOrder = [ - uplc-jvm.md -] diff --git a/site/docs/directory.conf b/site/docs/directory.conf index 457cccb9..8feeb8c6 100644 --- a/site/docs/directory.conf +++ b/site/docs/directory.conf @@ -10,8 +10,7 @@ laika.navigationOrder = [ schemes.md integrations extensibility.md - cookbook.md - demos + cookbook benchmarks.md quality-assurance.md migration-from-monocle.md diff --git a/site/docs/generics.md b/site/docs/generics.md index 22986a10..57e9c5c9 100644 --- a/site/docs/generics.md +++ b/site/docs/generics.md @@ -198,7 +198,7 @@ re-encode — or, with [eo-circe](integrations/circe.md) / [eo-avro](integration form in place and never fully decode at all. Put together, that turns a request handler into a short pipeline. The -**persist-and-stamp** recipe ([Cookbook → Effect threading](cookbook.md)) is the +**persist-and-stamp** recipe ([Cookbook → Effect threading](cookbook/effect-threading.md#persist-and-stamp-decode-free)) is the canonical shape — decode a PUT body into a domain object, store it effectfully, stamp the database-assigned id back on with a derived id-lens, and re-encode the result: the whole handler reads as `decode → store → stamp → encode`. @@ -332,7 +332,7 @@ The derived instance also backs `Plated.everywhere[S]` — a composable `Modify` that lifts any downstream optic to *every* depth, so `everywhere[S].andThen(prism).modify(f)` rewrites that focus across the whole tree. See [Optics → Modify](optics.md#modify) and the -[Cookbook recipe](cookbook.md). +[Cookbook recipe](cookbook/navigating-structures.md#visit-across-whole-trees). ```scala import dev.constructive.eo.generics.plate @@ -353,7 +353,7 @@ Plated.transform { case Expr.Var(n) => Expr.Var(n.toUpperCase); case e => e }( ``` Works on enums, sealed hierarchies, and recursive case classes. See -the [cookbook Plated recipe](cookbook.md) for a worked, runnable +the [cookbook Plated recipe](cookbook/navigating-structures.md#visit-across-whole-trees) for a worked, runnable example. (The block above is illustrative — the `new` the macro emits for an enum case can't run inside this page's doc sandbox; the cookbook runs the same derivation against a package-level diff --git a/site/docs/getting-started.md b/site/docs/getting-started.md index 33fc21aa..12007bfb 100644 --- a/site/docs/getting-started.md +++ b/site/docs/getting-started.md @@ -116,7 +116,7 @@ capability contracts across modules. Deep dive: - If you want the per-family tour with the kind of optic you'd reach for in each situation, read [Optics reference](optics.md). - If you'd rather start from a task — "edit one branch", - "batch-load nested IDs" — the [Cookbook](cookbook.md) opens with + "batch-load nested IDs" — the [Cookbook](cookbook/README.md) opens with a task index that maps straight to runnable recipes. - If you're coming from Monocle, jump to [Migrating from Monocle](migration-from-monocle.md). diff --git a/site/docs/index.md b/site/docs/index.md index 201a1799..69b6051d 100644 --- a/site/docs/index.md +++ b/site/docs/index.md @@ -113,7 +113,7 @@ library's core habit: [Capabilities](capabilities.md). - [Extensibility](extensibility.md) — how to ship a custom optic tuned for your domain's hot path without losing the rest of the cats-eo universe. -- [Cookbook](cookbook.md) — common patterns: option fields, +- [Cookbook](cookbook/README.md) — common patterns: option fields, composed Lens/Optional, multi-focus modify, JSON path edits, and `Plated` + `everywhere` — one optic applied at every depth of a tree, AST, or JSON document. diff --git a/site/docs/indexed.md b/site/docs/indexed.md index 644f1a4e..fda28b87 100644 --- a/site/docs/indexed.md +++ b/site/docs/indexed.md @@ -390,4 +390,4 @@ own residual label before reconstructing the outer structure. Indexed For variable-shape containers and aggregations, see [MultiFocus](multifocus.md). For the surrounding optic families, see the [Optics reference](optics.md#indexed); -for task-oriented examples, see the [Cookbook](cookbook.md). +for task-oriented examples, see the [Cookbook](cookbook/README.md). diff --git a/site/docs/integrations/avro.md b/site/docs/integrations/avro.md index 88508187..0f323b0c 100644 --- a/site/docs/integrations/avro.md +++ b/site/docs/integrations/avro.md @@ -1029,7 +1029,7 @@ For edits that recurse through every nested record rather than sitting at a fixed path — redact a field in every sub-record, rewrite a value at any depth — `Plated[IndexedRecord]` (from `dev.constructive.eo.avro.given`) makes the record tree a recursive -self-traversal. The [`Plated`](../cookbook.md) combinators +self-traversal. The [`Plated`](../cookbook/navigating-structures.md#visit-across-whole-trees) combinators (`transform`, `rewrite`, `children`, `universe`) then walk it, stack-safely. The immediate children of a record are its directly-record-valued fields; records nested inside array / map / @@ -1040,7 +1040,7 @@ Avro mirror of [`Plated[Json]`](circe.md). `Modify` you can `.andThen` a Lens / Prism onto so a single `.modify` rewrites that focus at every depth — composing exactly as it does for [`Plated[Json]`](circe.md). See the circe page and the -[cookbook recipe](../cookbook.md) for the runnable `everywhere.andThen(...)` +[cookbook recipe](../cookbook/navigating-structures.md#visit-across-whole-trees) for the runnable `everywhere.andThen(...)` shape. ## When to reach for which @@ -1064,7 +1064,7 @@ shape. | Render an Avro record as circe `Json` | `AvroJson.record(schema)` — `eo.avro.circe` | For the Kafka end-to-end recipe (read bytes, modify, re-emit), -see the [Cookbook → Serdes free pipes](../cookbook.md#serdes-free-pipes). +see the [Cookbook → Serdes free pipes](../cookbook/decoupling.md#serdes-free-pipes). For the full failure-mode matrix and the per-case behaviour specs, see [`AvroPrismSpec`](https://github.com/Constructive-Programming/eo/blob/main/avro/src/test/scala/dev/constructive/eo/avro/AvroPrismSpec.scala) diff --git a/site/docs/integrations/circe.md b/site/docs/integrations/circe.md index 47e50f7b..668a7a0e 100644 --- a/site/docs/integrations/circe.md +++ b/site/docs/integrations/circe.md @@ -441,7 +441,7 @@ every field named `ssn` at any depth, uppercase every string, round every number. `Plated[Json]` makes `Json` a recursive self-traversal — the immediate children of a node are an array's elements or an object's field values — so the -[`Plated`](../cookbook.md) combinators walk the whole document: +[`Plated`](../cookbook/navigating-structures.md#visit-across-whole-trees) combinators walk the whole document: ```scala mdoc:silent import dev.constructive.eo.circe.given @@ -486,7 +486,7 @@ everyString.modify(_.toUpperCase)(doc).noSpacesSortKeys to any depth (`transform` / `everywhere` on a call-stack/heap-machine hybrid, `universe` on a worklist, `rewrite` trampolined through `cats.Eval` so even a long re-fire chain won't overflow), so a deep -document is safe. See the [cookbook Plated recipe](../cookbook.md) for the +document is safe. See the [cookbook Plated recipe](../cookbook/navigating-structures.md#visit-across-whole-trees) for the data-type side of the same API. ## Working Json-first — migrating a Decoder-materialised model diff --git a/site/docs/integrations/jsoniter.md b/site/docs/integrations/jsoniter.md index 01838780..2dbcd875 100644 --- a/site/docs/integrations/jsoniter.md +++ b/site/docs/integrations/jsoniter.md @@ -447,6 +447,11 @@ per-element reassembly, no Composer hop required. | Edit deeply through `[*]` on the wire | Phase-3 (not yet shipped); fall through to eo-circe today | | Chain a JSON traversal with a Scala collection traversal | `JsoniterTraversal[A].andThen(Traversal.each[F, A])` | +For eo-jsoniter in a streaming service, see the +[allocation-free stream triage demo](../cookbook/grpc-filter.md): an fs2 filter that +classifies and filters 64-field messages straight off a socket, reading three +values per message instead of decoding it. + For the full failure-mode matrix (decode failures silently dropping to Miss, malformed JSON, length-changing splice mechanics) see the [`JsoniterPrismSpec`](https://github.com/Constructive-Programming/eo/blob/main/jsoniter/src/test/scala/dev/constructive/eo/jsoniter/JsoniterPrismSpec.scala), diff --git a/site/docs/migration-from-monocle.md b/site/docs/migration-from-monocle.md index 46f8be54..6981b476 100644 --- a/site/docs/migration-from-monocle.md +++ b/site/docs/migration-from-monocle.md @@ -277,7 +277,7 @@ reads on a worklist, `rewrite` trampolined through `cats.Eval`. cats-eo also exposes `everywhere[S]` as a composable `Modify` — `everywhere.andThen(prism).modify(f)` applies an ordinary optic at every depth, a combinator Monocle's Plated chooses not to expose. -See the [cookbook recipe](cookbook.md). +See the [cookbook recipe](cookbook/navigating-structures.md#visit-across-whole-trees). ## Cheat sheet diff --git a/site/docs/multifocus.md b/site/docs/multifocus.md index 41db8713..9d2bf47f 100644 --- a/site/docs/multifocus.md +++ b/site/docs/multifocus.md @@ -185,7 +185,7 @@ inner, without introducing a generic Glass Composer bridge. ## Worked examples -The [Cookbook](cookbook.md) shows container aggregation, batch-relative +The [Cookbook](cookbook/navigating-structures.md#compute-aggregations) shows container aggregation, batch-relative rewrites, and an `Indexed` environment-to-path example. For chains that continue through collections, use `Traversal.each` / `pEach`, including `Lens → each → Lens` and recursive `Plated` traversals. diff --git a/site/docs/optics.md b/site/docs/optics.md index 61ce966b..4296b4eb 100644 --- a/site/docs/optics.md +++ b/site/docs/optics.md @@ -348,7 +348,7 @@ typeclasses `F` admits (`.modify` for `Functor`, `.foldMap` for See the [MultiFocus reference](multifocus.md) for the full typeclass-gated capability matrix and composability profile; the -[Cookbook](cookbook.md) ships runnable recipes for the Grate, +[Cookbook](cookbook/navigating-structures.md) ships runnable recipes for the Grate, Kaleidoscope, and PowerSeries shapes. ### PowerSeries @@ -367,7 +367,7 @@ sweep sizes 4 / 32 / 256 / 1024). / `everywhere` — rides this same `MultiFocus[PSVec]` carrier via `Traversal.selfChildren`; it's a typeclass over the carrier, not a new family node. See [Generics → `plate[S]`](generics.md), the -[Cookbook](cookbook.md), and the [Modify section](#modify) for +[Cookbook](cookbook/navigating-structures.md#visit-across-whole-trees), and the [Modify section](#modify) for `everywhere`. ```scala mdoc:silent @@ -408,7 +408,7 @@ every focus to a single value with `.collectMap` (Functor-broadcast) or `.collectList` (List cartesian). Reach for it when you want to read the foci out as one summary rather than rewrite them in place. See [MultiFocus reference](multifocus.md) and -[Cookbook → Recipe B](cookbook.md). +[Cookbook → Recipe B](cookbook/navigating-structures.md#compute-aggregations). `collectList` supplies a singleton **focus vector** with observed context, not a guaranteed singleton source. Shape/count-coupled @@ -610,7 +610,7 @@ everyLeaf.modify(_ + 1)(Tree.Branch(Tree.Leaf(1), Tree.Branch(Tree.Leaf(2), Tree `ModifyF` (Lens / Prism / Optional / Modify), and the `.modify` runs bottom-up, stack-safe to any depth. For the read side (every sub-term as a list) use `Plated.universe`; for the full worked Prism-composition -recipe see the [Cookbook](cookbook.md), and for the macro that derives +recipe see the [Cookbook](cookbook/navigating-structures.md#visit-across-whole-trees), and for the macro that derives the `Plated` see [Generics → `plate[S]`](generics.md). ### Review diff --git a/site/laika-static/_redirects b/site/laika-static/_redirects new file mode 100644 index 00000000..9f8efe24 --- /dev/null +++ b/site/laika-static/_redirects @@ -0,0 +1,5 @@ +# Cloudflare Pages redirects for pages that moved: the cookbook was one page and is now a +# section, and the demos moved into it. +/cookbook.html /cookbook/index.html 301 +/demos/uplc-jvm.html /cookbook/uplc-jvm.html 301 +/demos/grpc-filter.html /cookbook/grpc-filter.html 301