Instrument · kernel autopsy Ⅱ

073: the invariant waste is dead — one tax remains

/alu/072 predicted what killing the pointer rebuild and gathers would reclaim. This is the follow-up with the measurement in hand: queries transposed once per tile into word-planar stack rows, the doc loop rewritten as AVX-512 intrinsics — and the emitted assembly re-read for what to do next.

1471

QPS @ 10M, C=2000, recall 0.974 (074, batch=32)

1685

QPS @ C=500, recall 0.922

vpgatherqq in the funnel (was 4/doc)

T=32

new tiling optimum (was 8) — √(s/k) as predicted

fig. 1the dispatch ledger — same x-scale as 072's 79-uop budget

42 uops/doc · ≈5.3 cyc/doc · 0.66 cyc/comparison

hamming math + data movementanswer-format tax (store-out)loopreclaimed vs 072's 79-uop budget

Hover a segment. Same x-scale as the 072 ledger — the dashed region is the budget 073 reclaimed. The red that remains is the last answer-format tax in the pipeline.

The math came out exactly as designed: doc words broadcast from the stream, query rows as folded loads at compile-time offsets — the stack transpose means 0x00/0x40/0x80/0xc0 are the addressing, no pointers to distill, nothing to gather. LLVM even arranged the adds as a pairwise tree.

But one red block survived, and it's new: the store-out. The kernel finishes with 8 complete distances in one zmm — and then disassembles that register lane by lane: 8 extract/store pairs, each with its own tail-guard branch, ~24 uops un-vectorizing what the math just vectorized. The same answer-format law from 072, one stage later: the cost was never the arithmetic — it's the format the next consumer demands. 074 shipped the two-instruction fix: vpmovqd narrows the lanes, one wide store writes the group — and the measurement confirmed the prediction: +13% (1307 → 1471 QPS at C=2000), with C=500 reaching 1685 QPS. Toggle the ledger to see the measured shape.

fig. 2the kernel arc — measured, recall bit-identical throughout

067 — generic loop650 QPS

069 — fixed-width unroll847 QPS

071 — typed + both-length gate955 QPS

072 — unsafe cast → LLVM re-vectorizes1093 QPS

073 — word-planar groups (T=32)1307 QPS

074 — wide store (vpmovqd)1471 QPS

C=2000, recall 0.9737 — bit-identical at every step. 2.26× from kernel work alone; recall never moved. Each step was one change, one measurement, one objdump.

Two things this arc teaches. First, the detour that didn't ship: written as safe scalar code, the planar loop scalarized — LLVM's popcount-idiom matcher doesn't recognize the shape it happily built itself in 072 from a gather pattern (822 and 736 QPS on the way to 1185). The kernel is intrinsics now; 012's “stay out of the autovectorizer's way” ends with an asterisk — when the pattern-matcher can't see your shape, you write the instructions yourself.

Second, the tiling knee moved 8 → 32 exactly as √(s/k) predicts when per-query tile state collapses from pointer machinery to 32 shared bytes in zmm rows. The scan now pulls ~44 GB/s of code stream at C=500 — within sight of the per-core streaming ceiling (068). The next factor of two lives in bytes or CCX-resident shards, not uops.

fig. 3the emitted hot loop (objdump a0eac–a0fe5)

vpxorq 0xc0(%r8,%rdx,1),%zmm0,%zmm0qw[3][0..8] ^ doc.w3 — the offset IS the transpose; no gather

vpxorq 0x80(%r8,%rdx,1),%zmm1,%zmm1qw[2] ^ doc.w2 (0x00/0x40/0x80/0xc0 = compile-time rows)

vpopcntq %zmm0,%zmm08 lane-counts for w3

vpopcntq %zmm1,%zmm18 lane-counts for w2

vpaddq %zmm0,%zmm1,%zmm0w3+w2 — pairwise tree, shorter dep chain than serial

vpxorq 0x40(...)/vpxorq (...) …w1, w0 same shape; final vpaddq → zmm0 = 8 complete distances

QPS measured on c8a.4xlarge (Zen5, 16 vCPU), 10M Snowflake arctic-256, exact ground truth, 3 reps (CV ≤ 3.5%). Uop counts read from the disassembly; the 074 column started as a prediction — the experiment existed to falsify it, and the measurement confirmed it instead at 1471 QPS (C=2000) and 1685 QPS (C=500).