Conversation
jsmucr
marked this pull request as draft
September 16, 2026 12:24
Member
|
Wouldn't this have been better as an issue for the moment? |
Records what the current tree does with Panasonic RW2 lens correction, identifies the regression introduced by 47c2237 (auto-selecting the embedded-metadata correction method for RW2, which is distortion-only and drops TCA), and outlines the follow-on work to decode the CA data present in Exif.PanasonicRaw.0x011b. Working document for the follow-on agent, not a permanent design doc.
Rigo's parseca.c four-checksum layout for 0x011b verified against the sample RW2 (all four match). Homeister's ExifTool forum 9366 post (retrieved via web.archive.org) supplies the interior structural map: 4-radial-zone model with N1..N4 at words [11],[4],[16],[17], on/off flag at [14], selector at 0x011a. The sample's ratios (1.0, 0.833, 0.667, 0.333) match the MFT-sensor pattern he reported. Coefficient roles within Homeister's three interior "parts" are still undecoded and remain the RE gap. Doc restructured to separate verified structure from remaining unknowns, prunes the earlier "two 16-word blocks" hypothesis that we now know is wrong, updates the image.h struct sketch to zone-based, and adds a shooting recipe for the RW2 corpus available on the developer's machine.
Log the outcome of the coefficient-fitting stage. The piecewise-linear 4-zone hypothesis with the 8 smooth words as knot heights and a single global scaling factor does not fit the measured CA, even with all knot permutations and a per-file gain: aggregate residual RMS 0.024 to 0.036 px against a null RMS of 0.037 px, and the per-file gains flip sign three to six times across the nine files with a 50x range in magnitude. Monomial-in-r and Catmull-Rom variants also fail. Measurement improvements captured: parabolic peak fit on the gradient profile with a peak-dominance filter, bilinear profile sampling and a same-feature integer-peak check, which lowers the per-bin half-vs-half RMS on 5 of 6 file/channel pairs on the three test files. The aggregated point cloud is 37,381 edges across 9 files, dumped to /tmp/rw2_tca/measurements.npz. Word[8]'s anti-correlation with the sign of measured R-G in 9/9 files still holds and is the strongest single-word signal in the tag, but does not extend to a coherent zone-height model with the other seven smooth words. The section is honest about the negative outcome: the 8-smooth-words piecewise-linear model with a global k is not the decode. Alternative hypotheses (three per-channel distortion polynomials, word[2] as a lens-family key rather than a coefficient) are listed as candidates for the next attempt, alongside the corpus additions that would resolve the ambiguity (in-camera CA-toggle pair, distortion-toggle pair, SILKYPIX Pro null-corrected reference).
GX80 mirror corpus (nine files across the same lens/focal set as the G9, at f/5.6) passes all four checksums on every file and shows the same structural layout: 0x011a=2, word[14]=256 flag, 4-zone model with identical 0.333/0.667/0.833/1.0 knot ratios, empty 0x0119 on the L 45-150, high-order flag bits on the PL 12-60, flag=1 on the primes. The 4 radii scale with body sensor size: GX80 N1=2888 vs G9 N1=3276, ratio 0.882, matching the ratio of half-diagonals within 0.5%. The ratios themselves are body-invariant. Word[8]'s anti-correlation with the sign of measured R-G radial displacement replicates cleanly on the GX80: 9 of 9 files, no exceptions. The two G9 files that were sign-ambiguous due to small measured amplitude come out unambiguous on the GX80. Removes the 'different body' item from 'What would resolve it' since that experiment is now done, replaces it with a note on the missing in-camera CA/distortion toggles (verified against DVQP1406ZA and by hand on both bodies).
Session 3 tried to decode 0x011b coefficients by measuring geometric distortion (much larger signal than CA) against the paired camera JPEG. Two hard findings: - The L 45-150 JPEGs are NOT distortion-corrected. Camera does not apply either 0x0119 (empty on this lens) or 0x011b to the in-camera JPEG. Residual 0.15 px RMS is registration noise, so fitting 0x011b against L 45-150 measurements is meaningless. - On files where 0x0119 is populated, its Rigo polynomial predicts distortion shape well (R^2 0.75-0.96) but the required scale factor swings from +0.90 to +5.24 to -0.95 within the same 12-60 zoom sweep. The 12-60's flag byte word[7] has high-order bits (0xF0, 0xB7, 0x8D at 12/25/60mm) that Rigo and 47c2237 both discard; primes all have 0x0001 flag and k close to 1. Strong hypothesis: the high byte of word[7] encodes a per-focal-length strength scaler. Implications: 47c2237's distortion output on Leica DG 12-60 is likely mis-scaled by factors of 2-5 as a latent bug on top of the Lensfun-TCA regression. Decoding 0x011b is blocked pending 0x0119's full decode.
Session 4 tested H1-H5 for a hidden per-focal-length strength scaler in word[7]'s high byte and finds all five hypotheses fail. Fresh k_only measurements are stable across bodies for the same tag data, but no simple function of word[7] (linear, subfield, LUT, whole int16) predicts the k pattern (best R^2 = 0.15). Nor does any of the untouched 0x0119 words (word[2, 3, 6, 9, 10, 13]) in single or two-way linear combinations. The wild k=+5.24 (25mm) and k=-0.95 (60mm) on the PL 12-60 that motivated the search are measurement artifacts of translation-only SIFT registration on files where the tag's polynomial predicts small amplitude (peak 2 px and 15 px). A joint fit of polynomial plus linear-in-r residual collapses k toward 1 on the eight strong-signal files (median 1.02) and shows a small (~0.5%) unmodeled radial dilation dominating the two anomalous files. Any model of the r-monotonic residual with a wrong-shape polynomial lands at a wild k. Practical consequence: session 3's implication (b) - that 47c2237's Panasonic distortion path may be silently mis-scaled - is retracted. The distortion math applies the tag's polynomial the way the tag intends. The default-method-selection regression covered elsewhere in this document is unchanged.
…0x011b words
Adobe DNG Converter emits a three-plane WarpRectilinear opcode on every
RW2 in the corpus, giving us Panasonic's own per-channel correction as
ground truth instead of pixel measurements. Fit D_R = R_plane - G_plane
and D_B = B_plane - G_plane against the smooth words in 0x011b.
D_R decodes cleanly: dropping w[2] (a per-lens family baseline), w[26]
and w[29] leaves six words that predict all four D_R.k_r{0..3} at
R^2_LOGO 0.97-0.99 under leave-one-group-out cross-validation, where
each held-out group is both bodies of a shared (lens, focal). D_B's
low orders (k_r0, k_r1) fit at R^2_LOGO 0.99; higher orders do not
generalize, because Adobe's B-plane coefficients differ 2-4x between
bodies at the same lens and focal even after LOGO training.
Section adds the ground-truth definition, sanity check against Rigo's
0x0119 polynomial (matches with a small per-file Adobe framing
constant), the LOGO scores of F1..F5 and reduced-word variants, the
6x4 coefficient matrix for D_R and the 6x2 low-order-only matrix for
D_B, and a stub of the follow-on darktable patch shape. No code
changes.
Session 6 runs the honest cross-corpus test session 5 asked for. 16 RW2 samples from DC-S5, DC-G9M2, DC-GH5 and DMC-GX8, sourced from raw.pixls.us, none of which was in the G9+GX80 training corpus. Structural findings (Rigo checksums, word[14] flag, Homeister radii) survive every third-body sample. Two structural corrections fall out: DC-S5 uses a full-frame-specific zone-radii pattern (1.0/6-7/4-7/2-7, not the MFT 1.0/5-6/4-6/2-6), and DC-G9M2 ships 0x011b without an 0x011a selector at all. Both matter for the darktable parse. The R-plane C_R decode reproduces the sign of the R-vs-G displacement on 69 of 78 signal-carrying bins across the four new bodies -- the same match rate the same pipeline gives on the training corpus. It over-predicts absolute pixel magnitude by ~10x on every body, including on the training corpus itself, so this gap is a property of Adobe's DNG opcode vs raw sensor CA and not a failure of C_R to generalize across bodies. Retested session 5's B-plane high-order gap: no single or pair of discrete words closes it. Adding four discrete words to the base six lifts D_B.k_r3 LOGO R^2 from 0.08 to 0.854 (marginal), but D_B.k_r2 tops out at 0.774. At 10 predictors on 18 files this is closer to overfit than a decode; the gap is still open.
Session 7 pairs each RW2 in a six-file subset (Sigma 16 and PL 12-60 at 12mm and 60mm, on G9 and GX80) with its camera JPEG, coarse-registers raw to JPEG with a partial affine, and measures R-G and B-G radial displacements in both with the parabolic sub-pixel fit already used against the raw. The camera JPEG carries essentially no CA (jpeg_RG within +/- 0.02 px on every signal bin), so the correction the camera applied equals the raw's own CA magnitude. Comparing that applied correction to the C_R prediction: - median-of-per-file-medians ratio = 11.48 - range 6.18 - 14.07 across six files (2.28x spread, cv = 30%) - per-body median: G9 = 13.29, GX80 = 8.05 - sign matches on 26 of 26 signal bins - direct read of Adobe's WarpRectilinear opcode (without the C_R fit) gives the same 4x - 30x ratios, confirming the gap is Adobe's opcode vs the raw, not a C_R decode artifact Outcome is the session brief's 'systematic mismatch (3-10x)' branch. A single global scale of 1/11.48 is right on the order but leaves per-file error at about 2x either way. Recommend not shipping C_R naively; understand Adobe's opcode magnitude first. Report goes in as a new subsection under 'Coefficient decode results'. No code changes.
Session 7 measured that Adobe's WarpRectilinear opcode over-corrects
the CA the camera JPEG actually applies by ~11x on a six-file subset.
Session 8 extends that measurement to the full 18-file corpus, then
asks whether the session-5 decode is structurally correct (same
coefficients, wrong scale) or whether it needs refitting against the
JPEG-derived applied correction.
Two refit shapes were tried against 207 (file, radial-bin) target
points:
- per-file poly4 fit then linear regression against words, mirroring
session 5. LOGO R^2 is negative on all eight polynomial
coefficients because the applied correction (0.01-0.15 px) is too
small to condition four polynomial degrees of freedom per file
against per-bin measurement noise.
- direct 24-DOF pixel-space fit on all bins at once. Train R^2 = 0.808
on R and 0.530 on B; under LOGO both go strongly negative (-6.11
and -0.99) and per-file RMS degrades to 0.38 px on the worst file.
Ridge does not help.
The scale factor generalises. Dividing session-5 C_R output by
K = 11.48 delivers per-file median RMS 0.028 px on R and 0.021 px on
B, worst-case 0.058 px and 0.034 px, with 96% of R and 75% of B
signal bins matching sign of the measured applied CA. Per-body K is
G9 10.4 / GX80 7.3, showing real body-dependent spread inside the
[7, 13] range, but no in-file signal keys on body cleanly. Ship
K = 11.48 constant.
Adds a JPEG-referenced decode section under 'Coefficient decode
results' and a JPEG-target patch formula block. Session 5's
Adobe-DNG-referenced matrices stay as historical reference with a
pointer to the session 8 K = 11.48 divisor.
Scripts at /tmp/rw2_tca/step19_{01..07}_*.py.
Static analysis of SILKYPIX Developer Studio 8 SE's SILKYPIX64.dll to
try to lift Panasonic's own algorithm for reading Exif.PanasonicRaw.0x011b.
Records what was recovered and what the analysis could not reach.
Recovered:
- the MSVC RTTI chain used to find IslZTiffExifPanasonic and
IslBRawReadPanasonic0 vtables, and the seven vtable slots that
Panasonic uniquely overrides,
- the class-field offset each Panasonic private-IFD tag maps to, from
the tag->stub table at RVA 0x17accc4. Tag 0x11b lives at
obj + 0x129a8,
- the five functions in .text that touch obj + 0x129a8 (via .pdata
RUNTIME_FUNCTION entries for accurate bounds), including the 88 KB
private-IFD parser at 0x4f3ca0..0x50950d,
- explicit no-hit results for the candidate CA constants (K = 11.48,
the Homeister zone radii {0.333, 0.667, 0.833, 1} in both MFT and
full-frame-S forms).
Not recovered:
- the exact word subset and polynomial form the parser applies. The
parser is 88 KB of MSVC-optimised code with the tag switch flattened
into an indirect-jump table, and lifting it further needs a
decompiler,
- the DefaultLensInfo.spd and DefaultParameters.spd container format,
which is a proprietary 'ISL Multi purpose file format'. Suspected
home of a per-lens attenuation that would explain the K factor,
but not confirmed.
Conclusion: session 8's decode is not refuted. Ship it as is. No
body-conditional scaling in the darktable patch; the per-body K spread
seen in session 7 probably lives in the .spd data, not in code.
Notes doc gets the understanding, not the decompiled listing. Nothing
from SILKYPIX enters the tree.
Session 9 stopped at the 88 KB private-IFD parser because linear disassembly could not lift its body. This session got Ghidra 12.1.3 headless working against SILKYPIX64.dll and used it to answer, for each of the five session-9 questions, what the binary is able to say. Two settled, two answered with strong negative evidence, one open. Session 8's decode still ships. New empirical findings: - The polynomial is not in the private-IFD parser. All nine touches of obj+0x129a8 in FUN_1804f3ca0 are reset/failure paths (memset the CA container to zero on error). - The CA container is 5512 bytes (0x1588), not the 64-byte payload we thought. It has sub-blocks and cached derived state, mapped partially from the class constructor at 0x4f07a0. - SILKYPIX does not compute csum mod 0xFFEF. The modulus appears three times in the whole DLL, none in Panasonic code. Either Rigo's checksum is not the validator SILKYPIX uses, or SILKYPIX trusts the tag unchecked. Recommendation: keep parsing the tag even if Rigo's checksum fails; log-only. - IslEISDevelopDemosaicPanaCA is a plugin, not a C++ class -- no RTTI type descriptor, no static factory table pairing class id 0x1216 with a constructor. The tag-0x11b decoder and polynomial evaluator live inside a factory-instantiated module whose entry point is not reachable from static class-hierarchy walks. - Property id 0xa052 = "ColorAberration" is the pipeline hook. Dispatched via FUN_18074e0f0 -> FUN_1810ceb30(..., 0xa052). That is the trailhead for a future dynamic-tracing session. The section captures which Ghidra scripts were written, which files were produced, what the reconciliation with session 8 amounts to (session 8 stays; per-body K spread stays unexplained), and honest bounds on what remained un-lifted.
Lifted the CA polyline builder at RVA 0x10ceb30 (FUN_1810ceb30), called from the property-key dispatcher with prop_id=0xa052. It produces a 9-knot IslZCnvPolyLine sampled by the render side, not a polynomial. Knots come from payload short indices 1..7 (radii) and 8..14 (values), each adjusted by seven int32 offsets fetched from property 0xa020; radii are normalised by half the raw-image diagonal and values by 1000. Boundary knots are (0, 1) and (2, knot[6].y). Verified against x86 disassembly. Session 8's six-word set overlaps SILKYPIX's use of words 8, 10 and 12 (as knot values 1, 3, 5), but SILKYPIX reads all of payload[1..14] and none past 14, so words 20/23/27 are not directly consumed here. The K = 11.48 factor is not a code constant, but its order of magnitude matches SILKYPIX's implicit value scale (short/1000). The polyline representation is not equivalent to a quartic, which explains session 8's LOGO refit failing. The raw-tag-to-property mapping (inside the 88 KB private-IFD parser) and the R/B split on the render side remain unlifted, so session 8's polynomial with K = 11.48 stays the shipping recommendation. Recommends: when both are lifted, switch the darktable path to a monotone spline evaluator on seven knots.
Session 11 had a shape-of-code right but the label wrong. The property-id table at RVA 0x1a117d0..0x1a11a70 in SILKYPIX64.dll maps 0xa050 -> Distortion, 0xa051 -> ColorAberration, 0xa052 -> Shading. Session 10 read it shifted by one entry and session 11 inherited the shift. The 9-knot polyline that session 11 called the CA descriptor is the shading descriptor: the dispatcher at 0x74f148/74f204/74f244 calls the builder for 0xa052/0xa053/0xa054 which are Shading and its two camera-correction variants, not CA. Session 12 also verified that 0xa051 (ColorAberration) never appears as an immediate anywhere in .text and its name string has no LEA references from code, so SILKYPIX 8 SE reaches Panasonic's CA correction through neither the property-id nor the property-name path -- if it reaches it at all. Property 0xa020, which session 11 called a per-lens offset table, is named 'Encryption Key' in that same property-id table; its origin was not lifted. The 58 KB FUN_1804f3ca0 was decompiled fully (raising DecompileOptions.setMaxPayloadMBytes past the 50 MB default that had silently truncated session 10), and contains no immediate for any of the CA/shading/distortion property ids, and no immediate for tag 0x11b. Its argument shape is (handle, struct*, small_int) and its body accesses fixed struct fields rather than iterating IFD entries, so session 10's identification of it as the private-IFD parser is not supported by the code. Plugin id 0x1216 = IslEISDevelopDemosaicPanaCA is confirmed via the plugin-name switch at RVA 0x547756, but there is no static edge from that id to a plugin factory. The shipping recommendation is unchanged: keep session 8's polynomial fit. Do not port session 11's spline; it would replace darktable's imperfect CA with a wrong-object-class shading correction.
Add a 'For the implementing agent' section right after the immediate workaround, containing the concrete recipe another agent needs to write the darktable patch: files to edit, struct extension, exif.cc parser with the four-checksum validator (log-only per session 12), CA evaluation in _init_coeffs_md_v2 including full C_R and C_B_lo matrices and K_JPEG divisor, sensor-format handling, auto-select policy, testing guidance, release-notes draft, and honest bounds on the fit. Add a 'Optional TODOs (post-shipping)' section before References, listing G9 firmware RE, Adobe Camera Raw RE, B-plane higher-order decode, third-body K validation, SPD file format work, dynamic tracing paths, and fit-robustness expansion. None block shipping. Update the TL;DR to reflect the state after 12 sessions: decode converged with LOGO R^2 0.97-0.99 on R, max RMS residual 0.06 px, cross- body validated on 6 Panasonic bodies. The older 'Where the code needs to change' section from session 2 remains below as an audit trail.
jsmucr
force-pushed
the
rw2-tca-compensation
branch
from
September 18, 2026 04:29
2fad937 to
2b79ef9
Compare
Since 47c2237, Panasonic RW2 files with a populated DistortionInfo default to embedded-metadata correction, but the Panasonic branch of _init_coeffs_md_v2 wrote the same radial multiplier into all three RGB channels, silently dropping the Lensfun-based transverse chromatic aberration correction that used to apply. Decode Exif.PanasonicRaw.0x011b alongside 0x0119. The 64-byte payload is stored as 32 signed int16 LE on dt_image_correction_data_t::panasonic; Rigo's four-checksum is validated and logged on mismatch but not gated on, since SILKYPIX itself does not enforce it and some newer bodies write valid coefficients without a matching 0x011a selector. Six words at indices 8, 10, 12, 20, 23 and 27 feed a fitted 4x6 matrix that reproduces Adobe DNG Converter's WarpRectilinear R plane on top of a global 11.48 divisor calibrating the output against the in-camera JPEG. B goes through a 2x6 matrix on its two lowest-order terms; the higher orders did not decode from the six-word set and are left at the G-plane value. Also mirror the tag to Exif.Image.0xf11b on TIFF/DNG round-trip, in the same TIFF private range as the 0xf119 alias 47c2237 added for the distortion tag. Decode corpus was 34 RW2s across six bodies (G9, GX80, S5, GH5, G9M2, GX8) with WarpRectilinear ground truth from Adobe DNG Converter and magnitude calibration on the paired G9 + GX80 subset with camera JPEGs. Worst per-file RMS residual against the JPEG-measured CA is 0.058 px on R and 0.034 px on B, both under one Bayer super-pixel. Sign matches the JPEG on 96% of R signal bins. See RW2_TCA_investigation.md for the decode audit trail.
The prior commit's K = 11.48 divisor from session 8 of RW2_TCA_investigation.md was fit against a raw-vs-JPEG edge-position measurement that filtered edges to those with all three integer R/G/B peaks agreeing within +/- 1 pixel. That filter by construction excludes every multi-pixel-shift edge, i.e. the exact edges the CA correction is meant to fix. The surviving sub-pixel-edge subset gave "applied correction" values of only 0.02-0.14 px, and fitting K to that subset produced a correction 10-15x too small for real strong-CA edges. Field test on P1366392.RW2 (G9 + Leica DG 12-60 @ 14mm, outside the training corpus) confirms the bug: the darktable render with K = 11.48 leaves obvious multi-pixel magenta fringing on every branch silhouette against sky, while the camera JPEG on the same frame has clean edges. |R-G| and |B-G| channel differences show bright edge halos throughout the darktable render and essentially none in the JPEG. Adobe DNG Converter's WarpRectilinear opcode is a workable ground truth: on P1366477 (PL 12-60 @ 12mm, training corpus) the opcode prescribes +0.77 px R shift at r=0.5 and +1.81 px B shift at the corner, and session 5's C_R evaluation reproduces those values within a few percent without any K divide. Setting K = 1 makes the shipped patch apply the DNG-derived magnitude directly. Re-rendering P1366392 at K = 1 clears the magenta fringing to near-JPEG parity. Rename _pana_K_JPEG to _pana_K since the name no longer reflects the role, and update the comment block to record why. Add session 13 to RW2_TCA_investigation.md documenting the finding, the audit trail and the residual (undecoded B-plane higher-order terms) that this does not close.
Session 13's commit refuted K = 11.48 and the framing that went with it: no unexplained per-body K spread, no missing firmware constant to recover K from, and no ambiguity between a real ~10x factor and a measurement artifact - the factor was a measurement artifact. Three Optional TODO items still framed themselves around the refuted premises: - TODO 1 (G9 firmware RE) claimed extracting the firmware would "resolve the origin of the K factor" and "whether the per-body K spread is algorithmic or a measurement artifact". Session 13 answered both. Drop those bullets; keep the item, refocused on the still-open B-plane higher-order coefficients (k_r2, k_r3) and the polynomial-vs-spline shape question. - TODO 4 (Third-body K validation) was framed as "if a third body's K lands close to 11.48, ship as-is with more confidence". K = 1 is now what we ship; the third-body test is still worth doing, but as validation that C_R generalises at unit magnitude, not as a measurement of a per-body K. - TODO 5 (SPD file format) mentioned "the ~10x factor and the per-body K spread might live there". Neither exists anymore. Downgrade the item to "could explain residual per-lens variance at K = 1", flagging its lower priority since Adobe DNG (which C_R matches) does not appear to consult such an overlay. Historical session narratives (5-12) are left as they were - they are the audit trail session 13 refutes, and rewriting them would break the document's own logic.
Removed session 7's ±1-integer-peak filter, redid the raw-CA measurement via 128x128 tile phase-correlation across all 18 corpus files, and attempted to refit C_R/C_B (4 rows each) against the resulting direct raw-side shifts. Findings: - The user-reported residual on P1366392 (Leica DG 12-60 @ 14mm, G9) is a shape problem, not a magnitude one. With only C_B_lo (2 rows, session 5), the shipped B polynomial can only produce ~0.07 px of correction at r = 0.95, where the raw actually carries +2.47 px of B shift. Session 13's K = 1 fix already handled magnitude for R; the missing piece is the B channel's k_r2 and k_r3. - Refitting C_R and C_B against direct raw measurements does not generalise. LOGO R^2 flips from 0.97-0.99 (Adobe DNG target, session 5) to negative on most coefficients. The direct-measurement targets are noisier than Adobe DNG's smoothed opcode values, and the ground truth still has 3-4x cross-body variance on B's higher orders on the same lens and focal length. - A hybrid using corpus-median Adobe D_B.k_r2/k_r3 as constants does essentially nothing, because the median is near zero (Adobe's values span negative to positive by near-equal amounts). Verdict: the 18-file / 9-lens-focal / 2-body corpus does not support recovering B's k_r2 and k_r3. Either a third body with paired JPEGs (option 3 in the Optional TODOs, samples being requested at the time of this session) or firmware RE (session 13 declared not tractable) would break through. No code change; C_R, C_B_lo and K = 1 stay shipped as-is. cacorrectrgb remains the practical user-side workaround for the residual.
Contributor
Author
|
@TurboGit Might be, but currently I only take it as my private investigation open to feedback. :-) In a proper issue I'd be expecting to be told to switch back to the lensfun preset. However... I may need some more RW2+JPEG pairs for this, so perhaps a forum post would be adequate. |
First pass on a third-body corpus using pre-production Panasonic DC-S5M2 samples from photographyblog. Four paired RW2+JPEG files covering the LUMIX S 85 f/1.8, S 20-60 and S 14-28 lenses across 14 mm to 85 mm focal length. Positive: - 0x011b tag is present on all 4 files with count = 64, and all four Rigo 2011 checksums pass. The checksum algorithm generalises to a new body family unchanged. - Radii ratios N2/N1 = 6/7, N3/N1 = 4/7, N4/N1 = 2/7 match session 6's DC-S5 full-frame pattern verbatim. The 4-zone structural model session 6 verified on the older DC-S5 applies to the newer DC-S5M2 without change. - Physical CA magnitudes measured by tile phase-correlation are on the same order as MFT bodies. B shift at r = 0.85 sits at ~1 px across all three lens/focal groups, matching the P1366392 residual. - rawpy + our measurement pipeline handle the 6000x4000 full-frame sensor without code changes (halfdiag is per-file). Negative: - Combining the 4 S5M2 files with the 18-file MFT corpus and refitting the six-word regression degrades B decode further at r = 0.85 (LOGO R^2 goes from +0.52 on MFT-only to -2.70 on the 22-file combo). Four S5M2 files across three lens-focal groups is under the parametrisation boundary for a 4-coefficient B polynomial. - Sweep of predictor sets (6, 8, 9, 27, 32 words) on the same 22-file corpus at r = 0.85 confirms: more predictors makes things worse. Session 5's [8, 10, 12, 20, 23, 27] remains the best six-word set. The bottleneck is data size, not features. No code change. C_R, C_B_lo and K = 1 continue shipping from session 13. The batch confirms option 3 is the right path but the sample size needs to be ~4x larger before the B higher-order decode has any chance.
RW2_TCA_investigation.md session 16 grows the corpus from the 18 MFT files of session 5 to 132 files across 14 bodies (5 full-frame: S5M2, S1M2, S1M2E, S1RM2, S9) and 50 unique lens/focal groups. The new C_R (24 doubles) and C_B (24 doubles, up from C_B_lo's 12) come from a direct raw-CA centroid fit on the combined corpus, not from Adobe DNG WarpRectilinear as session 5 did. Adobe DNG remains a useful reference for R; the B plane is where the DNG-based fit under- corrected, and where the raw-CA fit gains the most. With ~50 lens/focal groups constraining the polynomial, B's k_r2 and k_r3 -- which session 5 could not decode from six words on 18 files -- now decode cleanly under LOGO. B is shipped as four rows now instead of two; the Panasonic branch of _init_coeffs_md_v2 evaluates d_b = db_k[0] + db_k[1] r^2 + db_k[2] r^4 + db_k[3] r^6. K stays at 1.0. Function-space LOGO R^2 on the 132-file corpus (r = 0.30/0.50/0.70/ 0.85): R +0.72 / +0.70 / +0.57 / +0.14 B +0.59 / +0.65 / +0.52 / +0.39 Corpus-wide RMS residual at r = 0.85 drops from 0.79/1.20 px (shipped) to 0.66/0.52 px (new) for R/B -- better than 2x on B at the corner. Held-out on P1366392 (the user's reported strong-CA frame, PL 12-60 @ 14mm), the B residual at r = 0.85 drops from +1.31 px shipped to +0.70 px new, roughly 45% of the gap closed. On the rendered corner crop the mean |B-G| drops from 0.041 to 0.035 and the mean |R-G| drops from 0.054 to 0.038. No RELEASE_NOTES.md entry: the prior CA restoration entry for this cycle covers the feature, and this is a numeric refinement.
This reverts commit 3158eff (and its follow-up 2e04bf7). The refit's function-space LOGO R^2 at r = 0.85 (R = +0.14, B = +0.39) sounded positive but only 39% of the B corner variance is explained by the six-word predictor, and R at the corner is worse still. On the training corpus with strong CA that leaves the mean residual small - which the RMS table in session 16 flattered - but on files with genuinely weak CA the 4-row B polynomial extrapolates past LOGO's r-range with wrong-sign coefficients, so the correction introduces fringing where the shipped 2-row fit had left the pixel alone. Regression evidence: on P1366399 (Leica DG 12-60 @ 24 mm f/5.6) B at r = 0.95 is measured at +0.83 px; shipped predicted +0.03 (no visible action, correct); refit predicts -0.18 - the wrong sign, and applied as a correction it pushes B further from the measured position by 1.01 px. mean absolute R-G on the user-reported crop rises from 3558 to 4419 (+24 percent) when the refit correction is applied. Corpus-wide: 84 B and 88 R fit_ok files have some r in {0.70, 0.85, 0.95} where the refit is >= 0.15 px worse than shipped or crosses zero with predicted magnitude > 0.3 px. Full-frame bodies (only 17 LOGO groups across 44 files) are particularly affected. Fixing this properly needs a shrinkage prior (ridge toward the 2-row shipped fit or toward zero) or a monotonicity constraint on B(r), not simply more data. The K = 1 magnitude fix from 6c5bf9e stays.
…ual test - all dead ends)
Two developer hypotheses tested, both settled. The six predictor words are continuous coefficients, not per-copy identifiers: they move monotonically with focal length on every zoom with a focal sweep. But about 30 of 32 words change when only the body changes, and the predictors still move 100-500 counts between frames with lens, focal length and aperture fixed, so the payload responds to an input the fit cannot see. Panasonic's scheme is not shared with anyone. Olympus ca[6] is one whole-image polynomial per channel against 0x011b's four zones, and no other project decodes the tag; Homeister's 2018 forum post is still the only public RE. This closes the "Olympus ca[6] correspondence" open question. Two follow-ups ran to a conclusion, both negative: the inner checksums at words [2, 7, 13] resist a Rigo-family search over ~6.3e10 tuples per word, and normalizing the predictors by the zone radii confirms the words are body-scaled yet leaves the fit on session 18's exact ceiling. Names the identifiability check as the one cheap test still open.
Two tests, and together they read differently from either alone. Bootstrapping the tile set puts per-file sampling noise at 0.04 px at r = 0.85, yet files whose six predictor words are bit-identical disagree by 0.24 px RMS on measured B shift, six times that. Read alone this caps any function of the six words and leaves the refits headroom. But a demosaic-free estimator built on the raw Bayer sub-lattices disagrees with the AAHD pipeline by about 1 px with 29% sign flips at r = 0.85, including a sign flip on both regression targets, with no scale or offset relation that would let one be recovered from the other. Separately, four corpus files have no AAHD tiles beyond r = 0.7, so their r >= 0.85 values are extrapolation reaching +14.6 px, and every comparison in sessions 17, 18 and 20 was scored at those radii. So the near-duplicate floor is most economically explained as scene-dependent estimator bias rather than a missing metadata input, and sessions 14-20 optimised against a target whose per-file bias is comparable to the signal. That accounts for the whole shape of the ceiling, including session 18's visible regression on P1366392 despite an analytically correct-sign prediction. The shipped configuration stands on session 18's rendered-pixel check against SOOC crops, which does not depend on the AAHD pipeline, and that is the currency any future attempt should be scored in.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WIP
Turns out that Claude may be very well capable of extracting the required knowledge from various sources and a couple of RW2 files processed with SILKYPIX.