rebuild cache on MapRef, fix cancellation defects

README.md

@@ -139,6 +139,125 @@ libraryDependencies += "com.evolution" %% "scache" % "<latest version from badge
 * Touch, despite its name, is not called after refresh.
 * expireAfterWrite, despite its name, is calculated from date of creation, not time of update.
 
+## Benchmarks
+
+The `benchmark` module holds a JMH benchmark of the cache operations under contention. One
+invocation is the whole workload, 8 fibers running 20000 operations each against a key space of
+10000, so the reported number is cache operations per second with the contention included.
+
+```scala
+// everything, around 10 minutes
+sbt "benchmark/Jmh/run"
+
+// one scenario, one flavor
+sbt "benchmark/Jmh/run -p flavor=partitioned .*getOrUpdateHitRandomKeys.*"
+
+// longer run when the defaults are too noisy to tell two numbers apart
+sbt "benchmark/Jmh/run -wi 5 -i 10 -r 5s"
+```
+
+`flavor` picks how the cache is put together: `single` is one unpartitioned `LoadingCache`,
+`partitioned` is `Cache.loading`, `expiring` is `Cache.expiring` with the expiration set far enough
+away not to interfere.
+
+The whole suite is kept under ten minutes, which is one warmup and five measurement iterations per
+scenario. That is enough to compare implementations or spot a regression, not to argue about a few
+percent, and some of the scenarios below are visibly noisy.
+
+### Results
+
+Two runs back to back on the same machine, 12 cores, JDK 25, Scala 2.13.18: the cache as of commit
+`7c9fa9f`, where the whole map sat in one `Ref[F, Map[K, EntryRef]]`, and the same cache after it
+was rebuilt on `MapRef`. Millions of operations per second, before to after, higher is better.
+
+| Scenario | single | partitioned | expiring |
+|---|---:|---:|---:|
+| `getOrUpdate`, insert distinct keys | 1.25 to 2.06 (1.65x) | 2.11 to 2.41 (1.14x) | 1.91 to 1.99 (1.04x) |
+| `getOrUpdate`, hit random keys | 9.56 to 12.60 (1.32x) | 10.68 to 11.46 (1.07x) | 7.89 to 9.12 (1.16x) |
+| `getOrUpdate`, hit single hot key | 10.65 to 13.82 (1.30x) | 12.14 to 12.58 (1.04x) | 9.38 to 10.40 (1.11x) |
+| `get`, hit random keys | 21.74 to 26.10 (1.20x) | 21.37 to 23.02 (1.08x) | 12.68 to 13.22 (1.04x) |
+| `get1`, hit random keys | 19.47 to 23.66 (1.21x) | 18.78 to 22.14 (1.18x) | 12.59 to 12.92 (1.03x) |
+| `contains`, random keys | 26.69 to 31.97 (1.20x) | 23.59 to 33.70 (1.43x) | 23.22 to 33.68 (1.45x) |
+| `put`, insert distinct keys | 1.66 to 9.24 (5.56x) | 5.40 to 10.30 (1.91x) | 5.47 to 8.86 (1.62x) |
+| `put`, replace random keys | 8.13 to 9.53 (1.17x) | 7.37 to 8.83 (1.20x) | 7.88 to 8.00 (1.01x) |
+| `modify`, insert distinct keys | 1.88 to 11.44 (6.09x) | 6.13 to 10.65 (1.74x) | 7.15 to 11.76 (1.65x) |
+| `modify`, update random keys | 7.24 to 11.13 (1.54x) | 8.03 to 9.27 (1.15x) | 9.63 to 10.53 (1.09x) |
+| `remove` and `put`, random keys | 0.84 to 3.87 (4.59x) | 2.45 to 4.05 (1.65x) | 2.42 to 2.97 (1.23x) |
+| mixed `get`/`getOrUpdate`/`put`/`modify`/`remove` | 5.40 to 7.45 (1.38x) | 6.32 to 7.40 (1.17x) | 5.08 to 5.56 (1.09x) |
+
+The gains are largest exactly where the old implementation had to CAS the shared map, i.e. inserting
+and removing keys, and they shrink with partitioning, which is what partitioning was there to work
+around in the first place. Reads gain less, and `foldMap`, the one operation that used to get an
+atomic snapshot and now walks a `ConcurrentHashMap`, is a few percent slower: 1176 to 1146, 1142 to
+1105 and 1009 to 996 traversals of 10000 entries per second.
+
+Do not read too much into a single digit of these numbers. The suite is short by design, several
+scenarios have error margins of tens of percent, and the two runs were taken on a shared machine.
+The raw JMH output of both runs, error margins and all, is in `benchmark/results`.
+
+### Comparing against another revision
+
+The module builds against the `scache` sources next to it, so an older revision is measured by
+putting the module on top of that revision and running it there. Both runs have to happen on the
+same machine, one after the other, or the numbers are not comparable.
+
+```shell
+git worktree add /tmp/scache-old <revision>
+cp -r benchmark /tmp/scache-old/
+cp build.sbt /tmp/scache-old/build.sbt
+cp project/plugins.sbt /tmp/scache-old/project/plugins.sbt
+
+cd /tmp/scache-old && sbt "benchmark/Jmh/run -rf json -rff /tmp/old.json"
+cd -                && sbt "benchmark/Jmh/run -rf json -rff /tmp/new.json"
+```
+
+The `benchmark` project and the JMH plugin come from `build.sbt` and `project/plugins.sbt`, which is
+why those two are copied over as well. If the older revision has a different internal API, the
+benchmark will not compile there until the affected lines are adjusted. Going back past the `MapRef`
+rewrite, for instance, only the `single` flavor needs it:
+
+```scala
+case "single" => LoadingCache.of(LoadingCache.EntryRefs.empty[IO, Int, Int])
+```
+
+Finally, `git worktree remove /tmp/scache-old` when done.
+
+## Migrating to 7.0
+
+The cache state moved from a single `Ref[F, Map[K, EntryRef]]` to a per-key `MapRef` over a
+`ConcurrentHashMap`. What that means for the users:
+
+**Type classes.** `Cache.loading`, `Cache.expiring`, `SerialMap.of` and `SerialMap.apply` now ask
+for `Async[F]` instead of `Concurrent[F]` / `Temporal[F]`, because the new state needs `Sync` for
+the `ConcurrentHashMap` next to `Concurrent` for the fibers. Nothing to do for `IO` or for any stack
+that already has an `Async` instance, otherwise the call sites have to provide one.
+
+**Removed.** `LoadingCache.EntryRefs`, and with it the overload
+`LoadingCache.of(map: EntryRefs[F, K, V])`. Both were `private[scache]`, so this only affects code
+inside this library. `LoadingCache.of[F, K, V]` replaces them.
+
+**No more contention failures.** `getOrUpdate` used to give up with
+`IllegalStateException("extreme contention")` after 10000 lost CAS attempts on the shared state.
+Operations on distinct keys no longer contend at all, so the limit is gone along with that failure
+mode.
+
+**Cancelling a load cleans up.** Cancelling `getOrUpdate` now removes the entry it installed and
+fails everyone waiting for that entry with `CancelledError`, instead of leaving the key unusable and
+its waiters blocked forever. Note that the load is shared, so this reaches callers that were not
+cancelled themselves: if two requests ask for the same key, the first one runs the load and the
+second one waits for it, then a timeout cancelling the first fails the second with `CancelledError`
+as well. It gets to retry, where before it would have hung.
+
+**Loads can expire.** `ExpiringCache` evicts entries that have been loading longer than
+`Config.loadingTimeout`, failing their waiters with `ExpiredError`. The load itself is not
+cancelled, only detached from the cache. `loadingTimeout` defaults to the smaller of
+`expireAfterRead` and `expireAfterWrite`, so set it explicitly if the loads are legitimately slower
+than the expiration.
+
+**Enumeration is weakly consistent.** `keys`, `values`, `values1`, `size`, `foldMap` and
+`foldMapPar` are served by the `ConcurrentHashMap` and no longer observe an atomic snapshot of the
+map: an entry added or removed concurrently may or may not be included.
+
 ## Release process
 The release process is based on Git tags and makes use of [evolution-gaming/scala-github-actions](https://github.com/evolution-gaming/scala-github-actions) which uses [sbt-dynver](https://github.com/sbt/sbt-dynver) to automatically obtain the version from the latest Git tag. The flow is defined in `.github/workflows/release.yml`.  
 A typical release process is as follows: