A columnar text projection costs 3.4x the heap with the per-value copy excluded, and the remainder is in the decode path

[jdatcmd]

Found while separating width from column type for #766. It is not a costing defect: the
planner charges a text projection less than it charges the equivalent heap scan, and the
columnar scan then takes four and a half times longer.

Measured

4,000,000 rows, 14 columns, 2,000,197 rows returned, PG 17, min of 7 reps, serial plans on
both sides. The heap twin holds the same rows. The four text columns are
md5(i::text)-derived, 32 bytes each.

projection	columnar cost	columnar ms	heap ms	columnar ÷ heap
sel (one int4)	61,065	173.7	170.3	1.02x
n1 (numeric)	74,261	269.2	226.8	1.19x
n2 (float8)	73,196	351.5	232.0	1.52x
t1 (one text)	79,853	353.1	212.2	1.66x
t1, t2, t3, t4	136,218	1082.0	235.3	4.60x
eight int4 columns	138,522	271.2	220.6	1.23x
*	239,001	1297.6	186.2	6.97x

Read the two rows priced almost identically:

• eight int4 columns, cost 138,522, 271.2 ms
• four text columns, cost 136,218, 1082.0 ms

Nearly the same planner cost. Four times the real time. And against the heap, four text
columns cost the planner less than the heap's 158,109 while taking 4.6 times as long.

* is explained entirely by this: text accounts for 1082 ms of its 1298 ms, and all eight int
columns account for 271 ms. The earlier reading of * as a width result was wrong; it is a
type result.

Why this is its own issue

It is an execution finding, not a planner one. The costing makes it worse — a plan that
projects text is priced as if text were cheap, so it is chosen over alternatives that would be
faster — but correcting the cost model would only stop the planner choosing it. The 1082 ms
would still be 1082 ms.

It is also independent of #753 and #766. It needs no parallelism, and it reproduces on a
single serial scan of one column.

What is not yet established

• Which part of the text path costs it. Candidates, in the order I would test: the block
  codec (these are md5 strings, so compression is poor and the codec may be doing full work
  for little gain); FSST decode; per-value palloc and varlena construction in the output slot,
  which is where a columnar reader does work a heap reader does not (the heap hands back a
  pointer into the page, this reader must materialise each datum).
• Whether it is md5-specific. These strings are high entropy and 32 bytes. Low-cardinality
  text would take the dictionary path, and long text the TOAST path, and neither is measured
  here. The fixture was built to defeat compression, so it is the worst case for the codec
  and should not be reported as typical until a low-cardinality and a long-text cell exist
  beside it.
• Whether the heap comparison is fair on width. The heap stores the same 32-byte strings
  inline, so no TOAST fetch is involved on either side; that much is comparable. But the heap
  arm returns a pointer and the columnar arm materialises, and that difference is the thing
  being measured rather than a flaw in the comparison.

What would make this closable

A columnar text projection within a small factor of the heap's, or a measurement showing the
gap is inherent to materialising varlena datums from a decoded chunk and cannot be closed,
with the cost model then corrected to charge what it actually costs so the planner stops
preferring it.

[OffgridwithJD]

Took a perf profile at the first of the "not yet established" items — which part of the text
path costs it. Partial answer, plus one warning about how to measure this that cost me a
run and would cost the next person one too.

Container pgcolumnar-audit, pg18n (non-assert, so the profile is of real code), 1,000,000
rows, 4 md5 text columns and 8 int4 columns in one table, perf record -g attached to the
single backend running each query.

The warning first: do not use length() as the forcing function

My second probe used sum(length(t1))+… to guarantee each datum was materialised. Its profile:

19.91%  postgres  [.] pg_mbstrlen_with_len
 5.74%  postgres  [.] pg_utf_mblen_private

25.65% of that profile is UTF-8 character counting — length() on text walks the string
to count characters. That is the probe's own cost and has nothing to do with columnar decode.
Any measurement of "text projection is slow" that reaches for length() is measuring
length(). A wall clock taken that way (I got 367 ms text against 112 ms int) overstates the
storage-attributable gap by whatever that 25% is worth.

count(t1) is the better probe: the executor still has to deform the attribute, so the datum
is materialised, and count then only tests the null flag.

What the cleaner profile says

count(t1)+count(t2)+count(t3)+count(t4), top self-time:

 8.67%  libc         [.] __memmove_avx_unaligned_erms
 8.66%  postgres     [.] ExecInterpExpr
 6.81%  [kernel]     [k] clear_page_erms
 5.21%  libzstd.so.1.5.7
 4.18%  pgcolumnar.so [.] PgColumnarReadLogicalData
 3.89%  postgres     [.] AllocSetReset
 3.03%  [kernel]     [k] rmqueue_bulk
 2.61%  libzstd.so.1.5.7

against the same shape over eight int4 columns:

20.74%  postgres      [.] ExecInterpExpr
15.46%  pgcolumnar.so [.] pgcolumnar_row_read_column
 7.06%  pgcolumnar.so [.] bitunpack
 5.52%  pgcolumnar.so [.] PgColumnarDecodeChunk

Two things survive across both of my runs and I would treat only these as established:

• FSST does not appear at all, in either text profile, at any percentage above the 1.5%
  cutoff. Your second candidate is not where the time is going — at least not for md5 input,
  which is exactly the input FSST should refuse.
• zstd is real but not dominant: 7.82% combined. Your first candidate is doing work, and it
  is doing it on data it cannot compress, but it is not four-fifths of anything.

And one thing that did not survive, which I am flagging rather than reporting as a result:
clear_page_erms was 6.81% in the text profile of the first run but 8.53% in the int profile
of the second. So the page-clearing is not text-specific and I cannot use it to support the
per-value-allocation story, which is the candidate I expected to find and did not confirm.
memmove is consistently larger on the text side (8.67% / 11.44% against 7.80% / 3.15%), which
is suggestive of copying rather than decoding, but two runs is not a measurement.

What I would do next, and did not

The attribution wants a probe with no per-type cost of its own on either side. count(col) is
the closest I found, but the int arm and the text arm still differ in aggregate count (4 vs 8)
and in ExecInterpExpr share, so they are not directly comparable as profiles — only the
presence and absence of symbols is. A cleaner design would hold the aggregate identical and
vary only the column type, one column at a time: count(t1) against count(a), same row
count, same single aggregate.

That would also let the t1 row of your table — one text column at 1.66x — be attributed,
which is a better starting point than the four-column row, because it has the least of
everything else mixed into it.

I have not run the low-cardinality or long-text cells either. Your scoping note on those still
stands and this does not touch it.

[jdatcmd]

Attributed: a text column costs about 9x an int column at the margin, and the fixed scan cost was hiding it

The 4.6x in the issue body compares a four-column text projection against a heap. That mixes
the per-column cost with the fixed cost of scanning at all. Separating them makes the number
both larger and more specific.

First, an instrument that was void, because the same trap will catch the next person

The obvious design — hold the aggregate identical and vary only the column type,
count(t1) against count(a) — came back saying text is not slower than int:

column	type	min ms	vs int
a	integer	0.616	1.00x
t1	text	0.613	1.00x
n1	numeric	0.547	0.89x
n2	double precision	0.576	0.94x

All four at ~0.6 ms. The plan says why:

Custom Scan (PgColumnarScan)
  Output: (count(t1))
  Columnar Vectorized Aggregates: 1

count(col) on an unfiltered columnar table is answered from the vectors without decoding
the column at all. Every arm measured the fold shortcut rather than the storage. This is the
sibling of measuring sum(length(t1)), where a quarter of the profile turns out to be
pg_mbstrlen_with_len — the probe, not the subject.

Everything below therefore runs with pgcolumnar.enable_ungrouped_vector_agg = off and
enable_vectorization = off, and asserts the plan on every arm: the vectorized-aggregate
node count must be 0, with a control confirming it was 1 with the GUCs on. Turning something
off proves nothing unless it was demonstrably on.

Single column, fold disabled, type the only variable

column	type	min ms	med ms	vs int	avg_width
a	integer	171.7	181.1	1.00x	4
b	integer	167.2	176.4	0.97x	4
t1	text	353.5	359.1	2.06x	33
t2	text	351.6	355.4	2.05x	33
n1	numeric	249.5	254.8	1.45x	12
n2	double precision	289.4	293.0	1.69x	8

Both int columns agree to 3%, and both text columns agree to 1%, so the type is doing the work
and not the particular column.

The ladder, which is what the 2.06x understates

A single-column arm carries the whole fixed scan cost, so it dilutes the per-column
difference. Adding columns one at a time, same table, same rows, fold asserted off at every
step:

ncols	int, min ms	Δ	text, min ms	Δ
1	168.1	—	355.5	—
2	192.1	24.0	701.0	345.5
3	213.9	21.8	919.3	218.3
4	255.6	41.7	1160.4	241.1

Per additional column: int 29.2 ms, text 268.3 ms — a ratio of about 9.2x, taken as the
slope from the first to the last rung.

Stated honestly, the slopes carry real scatter: the three int deltas span 21.8 to 41.7 (a
factor of 1.9) and the three text deltas 218.3 to 345.5 (a factor of 1.6). And the two
extrapolated intercepts ought to agree, since the fixed scan cost is the same — they come out
139 ms (int) against 87 ms (text), which they do not. So 9.2x is the right order and should
not be quoted to two figures.

What this does to the issue's first open question

The question was which part of the text path costs it. This does not answer that, but it
moves the target and rules out one candidate by size:

• FSST does not appear in either text profile above a 1.5% cutoff — which is the expected
  behaviour for md5 input, exactly the input FSST should refuse.
• zstd is real but not dominant at 7.82% (5.21 + 2.61).
• clear_page_erms is not text-specific: 6.81% in a text profile against 8.53% in an
  int profile, so it cannot support a per-value-allocation story.

Roughly 268 ms per column against a codec share of ~8% means the codec is not where the time
goes. The interesting half is materialising a varlena datum from a decoded chunk — the
work a heap reader does not do, because it hands back a pointer into the page.

That is the next thing to measure, and it wants a design rather than a profile.

Profiling and the FSST/zstd/clear_page_erms findings are the reviewer's, including the
length() contamination that made them necessary. The count() shortcut was mine to walk
into.

[jdatcmd]

Addendum: the 9.2x understates, and by how much is derivable from the same table

With the aggregate path off the scan emits rows through the executor, so both arms pay the
same per-row deform and ExecInterpExpr. Every ratio above is therefore

(decode_text + shared overhead) / (decode_int + shared overhead)

which is closer to 1 than the decode-only ratio. It understates. That matters for how the
number is read: this measurement is smaller than the issue's original 4.6x for the
single-column arms, and a smaller number after a methodology fix looks like a smaller problem
when here it means the opposite.

The ladder gives the correction without another run. The int slope, 29.2 ms per column, is
almost entirely that shared per-column overhead — an int4 decode is close to free. So:

	ms per additional column
text, measured	268.3
int, measured (≈ the shared overhead)	29.2
text decode with the overhead subtracted	≈ 239

Against an int4 decode of ~0 after the same subtraction, so the decode-only ratio is not 9.2x
but very much larger, and 9.2x is the conservative floor. The honest statement is: a text
column adds about 239 ms per 2,000,197 rows that an int column does not, roughly 120 ns per
value, and the codec accounts for about 8% of it.

Both figures carry the scatter already stated: the int deltas span 21.8 to 41.7 and the text
deltas 218.3 to 345.5, so the subtraction is good to about ±20%, not better.

The understatement caution is the reviewer's.

[OffgridwithJD]

Two things on the assert-build contamination, one of which narrows the attribution and one of
which complicates the tidy version of it.

My profiles are not contaminated, and here is the check rather than the assertion

Both profiles I posted earlier ran on pg18n, and the naming on this box is a = assert,
n = non-assert. Confirmed from pg_config --configure:

pg15a pg16a pg17a pg18a pg18b pg18vg pg19a   ASSERT (--enable-cassert)
pg18n  pg18_san                              non-assert

and from the run artifacts rather than memory:

prof768.log   PG_CONFIG=/usr/local/pg18n/bin/pg_config
prof768b.log  PG_CONFIG=/usr/local/pg18n/bin/pg_config
AllocSetCheck occurrences in either profile: 0

AllocSetCheck at zero is the positive control — on an assert build it is 4–5% of the profile
and it would have appeared. So FSST absent above 1.5%, and zstd real at 7.82%, still stand.
Worth saying explicitly so those are not discarded along with the numbers that do need redoing.

The per-value allocation is structural, and it is narrower than "materialising a varlena"

PgColumnarDecodeValue (columnar_reader.c) has three branches:

if (att->attbyval)          { result = fetch_att(...); }              /* no alloc, no memcpy */
else if (att->attlen > 0)   { MemoryContextAlloc(attlen); memcpy(); }
else                        { MemoryContextAlloc(len);    memcpy(); } /* varlena */

int4, int8 and float8 are all pass-by-value, so the integer arms allocate nothing per
value. text and numeric are varlena and take one MemoryContextAlloc plus one memcpy
per value. On the 4-text-column shape that is ~8M allocations and ~8M copies against zero
for the integer arm.

That is also exactly why the assert build biased the ratio in the direction of its own
conclusion: AllocSetCheck is allocation-proportional and the integer arm has no allocations
for it to be proportional to.

It corroborates the profile too — memmove was consistently larger on the text side (8.67% and
11.44% against 7.80% and 3.15%), which is that memcpy. And it explains why clear_page_erms
did not discriminate for me: page-clearing is the OS handing over new pages as a context
grows, not the per-value allocation inside one that already exists.

But allocation does not explain the ordering, and I would not let the tidy version stand

sel   int4     attbyval -> no alloc      1.02x
n1    numeric  varlena  -> alloc+memcpy  1.19x
n2    float8   attbyval -> no alloc      1.52x
t1    text     varlena  -> alloc+memcpy  1.66x

If per-value allocation were the dominant term, both varlena rows would sit above both by-value
rows. They do not: numeric at 1.19x is cheaper than float8 at 1.52x, and numeric is the
one doing the allocating.

So allocation is necessary to the story and not sufficient. Something else separates float8
from int4 — both by-value, both fixed width, 8 bytes against 4 — and whatever that is also
applies to the text arm on top of its allocation. The 4-text-column figure mixes at least two
effects, and the single-column t1 row at 1.66x against n2 at 1.52x says the allocation is
worth something like the difference between them rather than the whole gap.

I would test it by holding width constant and varying only by-value-ness: int8 against
float8 (both 8 bytes, both by value) isolates nothing, but int8 against numeric sized to
the same bytes isolates the allocation. That is a cleaner discriminator than any of the arms
currently in the table.

[jdatcmd]

Decomposed: it is ~16% per-value and ~84% per-byte, and the codec is not the cause

Earlier work established that a text column adds ~232 ms per 2,000,197 rows over an int one.
That figure mixed two things PgColumnarDecodeValue does for every varlena — one
MemoryContextAlloc and one memcpy of the value's length — which scale differently. Holding
type, row count and encoding fixed and varying only the value width separates them without
ever leaving the varlena branch, so it does not reintroduce the encoding variable that already
cost this issue its ordering once.

Non-assert build (pg17_nc), 4,000,000 rows, encoding asserted NONE on every arm, fold
path asserted off.

width	encoding	min ms	Δ	ms per byte
8	0	198.3	—	—
16	0	240.0	41.7	5.21
32	0	331.8	91.9	5.74
64	0	545.2	213.3	6.67
128	0	935.4	390.2	6.10

Least squares over the five points: 6.20 ms per byte of width, intercept 142.7 ms, with
residuals of −9.2 to +6.0 ms — linear to about 2%.

An int4 control on the same cluster, which never enters the varlena branch at all: 105.3
ms.

The split, at the 33-byte width this issue's fixture uses

	ms	share of the excess over int4
total	331.8
excess over int4	226.5	100%
per-value varlena overhead (intercept − int4)	37.3	16%
width-driven	198.4	84%

So the per-value MemoryContextAlloc is 9.3 ns per value and about a sixth of the cost.
The other five sixths scale with bytes. Removing the allocation alone would not be the fix.

The codec is not the width-driven cost, by a second independent route

The obvious hypothesis was block decompression: against stored bytes the slope is
2.87 ms/MB, i.e. ~348 MB/s, which is squarely zstd territory, and against logical bytes it
is ~1.5 ns/byte — about 15x slower than memcpy bandwidth, a physical bound it should not be
able to cross if memcpy were the whole story.

Tested directly rather than reasoned about. Same data, same widths, compression => 'none'
against the default, with the block codec asserted from the catalog rather than assumed
(block_codec 0 against 3):

width	compression	stored MB	min ms
8	default	21.3	201.9
8	none	21.4	223.6
32	default	71.4	350.7
32	none	69.9	383.0
128	default	265.9	946.1
128	none	260.5	1066.6

Turning compression off makes it slower, not faster — 6.06 ms/byte against 7.10 — and the
stored size does not shrink. On md5 hex zstd achieves nothing and costs ~2% in size
(n128 at 260.5 MB is smaller than w128 at 265.9 MB), so what it is buying here is a
faster read path rather than fewer bytes.

That confirms the profiling result from the other direction: the codec was measured at
7.82% of profile with FSST absent above 1.5%, and now removing the codec entirely fails to
reduce the width-driven cost. Two independent methods, same answer.

What is left unexplained, stated rather than papered over

At 6.20 ms per byte over 4,000,000 rows the width-driven cost is ~1.5 ns per logical byte, or
~667 MB/s effective. That is slow for memcpy and fast for nothing else obvious. The
candidates are streaming ~280 MB from RAM with 4,000,000 small copies rather than one large
one, which is per-call-overhead bound rather than bandwidth bound — but that predicts a cost
flatter in width than what was measured, and the measurement is linear to 2%. I do not have
an explanation that fits both, and the next step is a profile of the width sweep specifically
rather than another hypothesis.

What this changes

• The fix direction is not "avoid the per-value palloc". That is 16%.
• Returning a pointer into the decoded chunk buffer instead of copying would address both
  components, which makes it the right shape — but it carries a lifetime hazard that ASAN
  cannot see (a MemoryContextReset use-after-free is invisible to it; the assert build's
  CLOBBER_FREED_MEMORY is the instrument), so it is not a change to rush.
• The ~232 ms per text column headline and the >= 9.05x floor are unaffected.

[jdatcmd]

The width profile, with the discriminator stated before the run — and it refutes the prediction

The candidate was: the cost is not the copy, it is acquiring and zeroing the memory to copy
into, so clear_page_erms and rmqueue_bulk should grow with width while __memmove stays
flatter. The opposite branch was stated too: if __memmove grows and the kernel symbols stay
flat, it is a genuinely slow copy path.

Profiled at 8, 32 and 128 bytes, non-assert, backend pinned, attached by PGAPPNAME.
Shares converted to absolute time, because a share of a growing total is not a measurement:

symbol	w=8	w=32	w=128
__memmove_avx_unaligned_erms	14.9 ms	38.5 ms	147.0 ms	grows 10x
clear_page_erms	0.4	2.3	12.6	grows, but tiny
rmqueue_bulk	0.0	0.0	0.3	negligible
AllocSetReset	24.7	23.8	26.0	flat
AllocSetAlloc	3.2	3.6	3.5	flat

__memmove grows and the page-allocator symbols do not. So the copy is real work that
scales, and AllocSetReset — which I had half-suspected too — is absolutely flat at ~25 ms
across a 16x change in width. Context reset is not width-driven.

And the copy is not slow after all

	bytes copied	in	rate
w=8	45.8 MB	14.9 ms	3.00 GB/s
w=32	137.3 MB	38.5 ms	3.49 GB/s
w=128	503.5 MB	147.0 ms	3.34 GB/s

A consistent ~3.3 GB/s, which is an ordinary rate for 4,000,000 small copies out of a
streaming source. The earlier "645 MB/s, 15x below memcpy bandwidth" figure was wrong because
it attributed the whole width-driven cost to the copy. The copy itself is fine.

So the open question narrows rather than closes

At w=128 the width-driven cost is 793 ms of the 935 ms total, and __memmove accounts for
147 ms of it — 19%. The remaining ~650 ms is not in any single symbol above the 0.8% cutoff.

What the profile does show is a cluster of kernel page-management symbols that is absent at
w=8 and present at w=128: ___pte_offset_map 3.61%, sync_regs 1.79%, clear_page_erms
1.35%, __rcu_read_lock 1.24%, zap_present_ptes 1.07%, memcg1_commit_charge 1.07%,
native_irq_return_iret 0.92% — about 11% between them, none individually large.

So the reviewer's instinct that kernel page handling scales with width is supported, while
the specific mechanism named (rmqueue_bulk then clear_page_erms) is not: it is the
page-fault and mapping path as a whole, consistent with cycling hundreds of megabytes per
query through allocation and reset. Rehabilitating clear_page_erms was the right call —
it does grow 34x — but it is 12.6 ms, not the story.

Still unexplained: ~650 ms at w=128 spread below the profile's resolution. The next step is
a profile with no percent limit and children aggregated by call path, not another hypothesis.
I would rather leave that stated than pick the plausible-sounding candidate, which is what the
last two rounds have each cost.

[jdatcmd]

The counters answered it, and then a lever I had to run twice named the mechanism

Recap of the discriminator you proposed, because the answer turns on the word marginal:

instructions/byte FLAT while cycles/byte GROWS -> the work is not growing, the STALLS are.
Memory-bound, and no symbol will ever show it. instructions/byte GROWS -> there IS more work
per byte [...] and --children aggregation will find it.

Counters on pg17_nc, backend pinned to cpu_core, PMU fraction >= 95% enforced per arm,
4,000,000 rows, encoding asserted NONE, fold off:

width    instr/byte  cycles/byte    IPC   cachemiss/kB   LLCmiss/kB   dTLB/kB
8            81.558       19.386   4.21         9.9235       0.0000     0.9016
32           36.561       11.062   3.31        38.7030       0.0000     1.1950
128          20.311        8.864   2.29        71.4136       0.0000     1.2434

Read straight down, instructions/byte falls, which looks like your first branch. It is not: that
column is the average including the fixed per-value cost, which amortises over more bytes as
the value grows. The quantity the slope is about is the marginal, and it is header-free --
instr/row is measured as ins/(rows*reps), so whatever the varlena header is, it cancels:

              marginal instr/byte   marginal cycles/byte   marginal IPC
  8 -> 32              14.06                 6.90              2.04
 32 -> 128             14.22                 8.04              1.77

14.1 instructions per marginal byte, flat to 1% across a 16x width change. That is your
second branch: real work, growing linearly with bytes. The stall term is present but secondary
-- holding IPC at the 8->32 value would predict 6.90 cycles/byte where 8.04 is measured, so
about 14% of the marginal cycles are added stalling and 86% is work. LLC-load-misses is zero
throughout, which is not a memory-bound signature.

Sanity bound, because 14 instructions/byte is a lot: memcpy moves 32 bytes per instruction.
This is not the copy. Something is touching each byte with roughly a dozen instructions, or
touching it a dozen times.

--children found it, exactly as you said it would

Recorded both widths, converted to absolute time rather than percent -- a percentage cannot
tell "grew" from "everything else shrank", which is the arithmetic that produced the 645 MB/s
claim I withdrew. Two independent instruments agree on the anchor: sample count / 997 / 13
queries gives 185.2 and 887 ms against timed 184.6 and 889.1 ms.

frame (children)              w=8              w=128
asm_exc_page_fault      11.50% =  20 ms   37.32% = 326 ms
  do_anonymous_page      3.16% = 5.5 ms   16.51% = 144 ms
  alloc_anon_folio       1.12% = 2.0 ms   13.30% = 116 ms
__memmove_avx...        10.93% =  19 ms   26.59% = 232 ms
AllocSetReset           12.62% =  22 ms    6.02% =  53 ms

Total growth is 700 ms and page faults are 306 ms of it. Counted directly rather than
inferred: 314,766 minor faults per query at w=128. That is 1.29 GB of anonymous memory
faulted in fresh for a 528 MB column, every single query.

The lever, and why the first run of it refuted nothing

Faults can only recur if something gave the pages back. glibc frees a large block by returning
it to the kernel, so raising MALLOC_MMAP_THRESHOLD_ / MALLOC_TRIM_THRESHOLD_ should stop it
without touching pgcolumnar.

My first run reported no change and I nearly published that as a refutation. It was inert.
pg_ctl was run as root, postgres refuses to run as root, the error went to /dev/null, and
the original postmaster served every arm. The only reason this is a redo rather than a wrong
published finding is that the script asserted the premise -- postmaster env carries MALLOC_MMAP_THRESHOLD_: 0 -- and refused the arm. A lever you did not verify pulled refutes
nothing.

Run as postgres, premise gating each arm:

w=128, 4,000,000 rows x 128-byte text, non-assert pg17_nc
baseline              923.936 ms         314766 minflt/query   (premise env=0 OK)
malloc-lever          591.967 ms           2244 minflt/query   (premise env=1 OK)
back-to-base          907.096 ms         314028 minflt/query   (premise env=0 OK)

Faults fall 140x and the query gets 332 ms faster, with a drift control that returns to
baseline on both columns.

What this means for the issue

The decomposition now reads: of ~793 ms of width-driven cost at w=128, ~332 ms (42%) is the
kernel faulting and zeroing pages that glibc handed back, reached from the per-row-group
allocations. That is why the symbol profile looked dispersed with no large leaf -- the cost is
in the fault path, entered from many allocation sites, and a leaf profile scatters it.

It also fits the counters: page-fault entry plus clear_page is instruction-heavy and scales
with bytes, which is a flat marginal instructions/byte at high IPC, not a stall.

The env var is the diagnosis, not the fix. The fix belongs in pgcolumnar: stop allocating
and freeing a large per-row-group buffer that crosses glibc's mmap threshold, and reuse one
across groups instead. I have not yet pinned which allocation, and I am not going to name a
candidate before measuring it -- that is what the last two rounds each cost. The next probe is
the second, independent lever: shrink chunk_group_row_limit so the per-group buffer drops
below the threshold on its own, with no env var. If the faults collapse, the allocation is
sized by the chunk group and the fix is a buffer that outlives the group.

Two corrections to earlier comments on this issue, both mine:

• "page allocation is refuted" was wrong. It was refuted as a leaf (AllocSetReset is flat,
  rmqueue_bulk negligible), and I took that as refuting the class. --children on the same
  data shows the fault path is the single largest growing term.
• the first malloc-lever result should be struck. It measured a cluster that never restarted.