Solomus

CHANGELOG.md

@@ -2,6 +2,32 @@
 
 All notable WIMF changes are recorded here. The project follows semantic versioning for the Python package; container compatibility is documented separately.
 
+## 2.3.0 - 2026-08-26
+
+- Predictive tiles gain an adaptive range-coded entropy stage (tile entropy
+  byte 2): signed residuals through the shared coefficient models with
+  per-predictor contexts, competing with Zstandard during scoring and stored
+  only when smaller. Noise-heavy content drops about a quarter in size.
+  The pure-Python decoder rejects byte-2 tiles with a clear message; the
+  native decoder reconstructs and validates them fully.
+- Wavelet lossy tiles use per-subband range-coder probability contexts
+  (reversible flag 6), worth about half a percent over flag 4. Lossless
+  stays on the single-context flag 3 stream where banding measured
+  net-negative. Flags 0-5 remain byte-compatible for old files.
+- Fixed a latent MinGW-only heap corruption at worker-thread exit:
+  thread_local Zstandard contexts wrapped in destroying unique_ptr raced
+  emutls teardown during pthread key cleanup. Contexts are now bounded,
+  intentionally leaked allocations. Found via stress hammering and
+  confirmed fixed under Dr.Memory with zero error reports.
+- Hardened range-coder decoding: consumption limits derived from payload
+  sizes turn hostile or desynced streams into clean rejections instead of
+  out-of-bounds reads past the container.
+- Pinned the quality ladder's rate-monotonicity with a regression test;
+  the historical Q1-larger-than-Q2 inversion no longer reproduces after
+  the 2.1/2.2 retunes.
+- Documented the progressive-layer reservation rationale in the WIM2
+  specification: `layers != 1` stays rejected pending the 3.0 container.
+
 ## 2.2.4 - 2026-08-25
 
 - Wavelet tiles now use an adaptive binary range coder (LZMA-style, with

docs/roadmap.md

@@ -40,8 +40,11 @@ section 5b target the largest one - compressed file size - first.
 - [ ] Native Android build support (Termux/NDK): runtime dispatch already
   resolves the inactive-NEON report in issue #31; document the Android Bionic
   Zstandard `qsort_r` build note for native builds.
-- [ ] Resolve progressive-layer design: either implement multi-layer coding or
-  publish the reservation rationale in the WIM2 specification.
+- [x] Resolve progressive-layer design: the reservation rationale is published
+  in `docs/wim2-format.md` (tiles are independently decodable, layer
+  bookkeeping would touch every container invariant at once, and true
+  embedded progressive streams move to the WIMF 3.0 container break).
+  `layers != 1` remains rejected.
 
 ## 3. Web and languages
 
@@ -103,13 +106,16 @@ section 5b target the largest one - compressed file size - first.
 - [ ] YCoCg-with-offsets refinement of the color transform remains open.
 - [x] Context-modeled entropy coding for wavelet subbands: an adaptive binary
   range coder (LZMA style, 11-bit probability models) replaces varint+zstd
-  payloads behind reversible flags 3/4; lossy Q5 harness went from
-  1.58x @ 25.48 dB to 2.31x @ 45.46 dB. Legacy unpackers retained for old files.
+  payloads behind reversible flags 3/4/6; lossy tiles add per-subband
+  probability contexts (flag 6), worth about half a percent on the bench
+  corpus, while lossless keeps the single-context stream (flag 3) where
+  banding measured net-negative. Legacy unpackers retained for old files.
 - [ ] Extend context-modeled entropy coding to prediction residuals;
   predictive and palette tiles still use generic Zstd payloads.
-- [ ] Rebuild the quality→quantizer ladder as a smooth, rate-monotonic curve;
-  today Extreme records 6.89× at Q1 versus 17.31× at Q2 across every tested
-  system, so lower quality currently produces larger files.
+- [x] Rebuild the quality→quantizer ladder as a smooth, rate-monotonic curve.
+  The 2.1/2.2 retunes eliminated the inversion (Extreme Q1 once recorded
+  6.89x versus 17.31x at Q2); verified rate-monotonic across flat, gradient,
+  and noisy content by `test_lossy_size_monotonic_in_quality`.
 - [ ] Optional lossy chroma decimation for photographic tiers, reconstructed
   during decode without changing the WIM2 container.
 - [x] Pin down the quality=10 contract: losslessness comes only from the

docs/wim2-format.md

@@ -28,18 +28,40 @@ Each tile record contains `x:u16`, `y:u16`, `width:u16`, `height:u16`,
 
 Payloads do not overlap semantically and each tile is independently decodable.
 Current modes are Raw (0), Predictive (1), Palette (2), and Wavelet (3);
-entropy IDs are None (0) and Zstandard (1). WIMF 2.2 writes and accepts exactly
-one layer. Other layer counts are reserved and rejected rather than silently
-misdecoded.
-
-Readers validate dimensions, tile coverage, mode and entropy IDs, offsets, expanded-size limits, metadata limits, and checksums before decoding. ROI decoding reads only intersecting entries.
+entropy IDs are None (0), Zstandard (1), and Range-coded (2, predictive
+residuals only). WIMF 2.2 writes and accepts exactly one layer. Other layer
+counts are reserved and rejected rather than silently misdecoded.
+
+### Why the layers field stays at 1
+
+The `layers` byte reserves room for quality-progressive coding: multiple
+refinement passes per tile that a decoder could stop after to get an early
+coarse image. Multi-layer coding was considered and deferred for WIM2:
+
+- Tiles are already independently decodable; a progressive client gets most
+  of the practical benefit by decoding tiles in priority order rather than
+  by partially decoding each tile.
+- Layer bookkeeping would touch every container invariant at once: per-layer
+  offsets and checksums in the index, AROT shard repair across layers, ROI
+  intersection semantics, and the raw-size validation limits.
+- The adaptive range coder with subband-aware contexts captures the size
+  wins that motivated layered refinement at a fraction of the complexity.
+
+True embedded progressive streams (zerotree or bitplane coding) are tracked
+as a WIMF 3.0 item, where the container break makes layer-native design
+cheaper than retrofitting this format. Until then `layers != 1` is rejected;
+the value survives so old files never need migration if 3.0 changes course.
+
+Readers validate dimensions, tile coverage, mode and entropy IDs, offsets,
+expanded-size limits, metadata limits, and checksums before decoding. ROI
+decoding reads only intersecting entries.
 
 ## Coding modes
 
 - Raw stores pixel bytes when codec overhead would increase size.
-- Predictive selects a spatial predictor per channel row and compresses reversible residuals.
+- Predictive selects a spatial predictor per channel row and compresses reversible residuals; residuals ship Zstandard-compressed or, when smaller, as an adaptive range-coded stream (entropy ID 2).
 - Palette stores up to 256 local colors plus one-byte indices.
-- Wavelet uses reversible CDF 5/3 for lossless data and quantized CDF 9/7 for lossy data. Coefficients use zero runs and zigzag varints before entropy coding.
+- Wavelet uses reversible CDF 5/3 for lossless data and quantized CDF 9/7 for lossy data. Coefficients are packed by an adaptive binary range coder (reversible flags 3/4 single-context, 6 with per-subband contexts); zero-run varint packing remains decodable via flags 0-2.
 
 `auto` classifies each tile to shortlist candidates. Lossless selection uses actual encoded size. Lossy selection combines encoded size and reconstructed distortion. The chosen mode is always recorded; decoders never classify.
 

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "wimf"
-version = "2.2.4"
+version = "2.3.0"
 authors = [
   { name="BenchWare", email="ivanm12453@gmail.com" },
 ]