Hive store layout (morton-hive/1, /2)

zagg can write each dispatch shard as its own self-describing leaf zarr under a morton digit tree, instead of into one shared flat store. The layout is the write-side half of the sparse-coverage design record (docs/design/sparse_coverage.md §2–§3, decisions D1–D6); the convention itself is owned by the mortie spec and versioned as morton-hive/1. A store that declares a time-window schedule (D13–D15, Time windows below) is morton-hive/2 — a strict superset: a /1 store is a /2 store with schedule: none.

It is the default for HEALPix output (issue #253) — point aggregation and the raster pipeline alike (issue #247; the hive/flat split sits one abstraction above the pipeline kind): an omitted output.store_layout resolves to hive. An explicit store_layout: flat (the single shared store) remains for interop/debug but is deprecated — removal is gated on the sparse-DGGS read path (issue #251 phase 3); rectilinear grids keep the flat shared store. Hive is wired to both backends — the local runner and the Lambda handler share the same per-shard write path (see Status).

Layout

{store_root}/
  morton_hive.json               <- static manifest (root-only exception)
  {sign+base}/{d1}/.../{d_n}/    <- one decimal digit per level (D2)
    {full_id}.zarr/              <- vanilla zarr v3 leaf, one per shard (D3)
    {full_id}_{window}.zarr/     <- time-windowed leaf (D13, morton-hive/2)
      zarr.json                  <- commit stamp (either leaf form); on a
                                    VERSIONED leaf it also names `current`
      {cell_order}/...           <- a legacy leaf's arrays, at the root
      run-{run_id}-{attempt}/{cell_order}/...
                                 <- a versioned leaf's arrays (own stamp)

• Ids are morton decimal strings (D1): sign + base digit (1..6 / -1..-6), then one digit per order, digits 1..4, never 0. A string prefix is the spatial ancestor, so cross-resolution containment is fine_id.startswith(coarse_id) — arithmetic, not I/O.
• One digit per path component (D2), so shards at mixed orders nest naturally: every order is a legal node.
• Full id at the leaf (D3): .../-5/1/1/2/3/3/3/-5112333.zarr is self-describing without parsing its directory chain — greppable in inventories, unambiguous if moved. Each leaf is a vanilla zarr v3 store: the same group/array template as the flat layout, sized to one shard (dense arrays hold cells_per_shard cells; resolution: chunk companions hold the shard's K inner chunks). When sharded (the K > 1 default, matching flat — issue #236) each dense array is ONE ShardingCodec object spanning the whole leaf, written at leaf block 0. A ragged field's vlen-bytes array is one whole-leaf object whenever K > 1 — sharded or not (issue #209), independent of the dense sharded toggle. The ShardingCodec is itself vanilla zarr v3, so the leaf stays self-describing. One recorded exception (issue #200, O8): the coverage.moc occupancy-bitmap sidecar inside the leaf — a single foreign key that zarr readers ignore (data reads are unaffected; member enumeration like members()/tree() emits a ZarrUserWarning and skips it).
• Node invariant (D5): below the root, a node contains only digit children ([1-4]/) and *.zarr objects — zero zarr metadata above the leaf, no shared mutable state across workers. The root alone also carries the manifest and the coverage sidecars — coverage.moc, plus coverage.toc on a temporal store. zagg.hive re-checks every computed leaf path against this invariant before writing.

Config

output:
  store: s3://bucket/product        # becomes the hive root
  store_layout: hive                # the HEALPix default (issue #253); may be omitted
  grid:
    type: healpix                   # hive is HEALPix-only (morton digit tree)
    parent_order: 9                 # shard order -> tree depth
    child_order: 13                 # cell order

sharded output (docs/sharding.md) is supported and is the default whenever chunk_inner gives K > 1 — same contract as flat (issue #236): each leaf's dense arrays collapse to one object apiece instead of K per-inner-chunk objects PUT onto a single leaf prefix. An explicit sharded: false opts the dense arrays back into K streaming objects — the ragged vlen array stays one whole-leaf object regardless (issue #209); an explicit sharded: true at K == 1 validates and is a no-op (nothing to bundle — the leaf is byte-identical either way).

Validation rejects hive with a rectilinear grid (node names are morton digits) and with consolidate_metadata: true (there is no store-root zarr hierarchy to consolidate — D5/D12). (The manifest's shard_order field below records the dispatch/tree order — it is not a config knob.)

Time windows (morton-hive/2)

A store may partition each shard's time series into one write-once leaf per window (issue #246, design D13–D15; grammar and boundary semantics frozen on the mortie spec page). Windowed leaves keep full D4/D5 semantics — stamped, binary debris, zero shared state — so backfill is just a new earlier-window leaf, concurrent runs on different windows share no object, and the window is the unit of idempotent reprocessing (re-dispatching a window replaces its leaf wholesale).

output:
  store_layout: hive
  windowing:                        # absent = schedule none = morton-hive/1
    schedule: yearly                # none | yearly | monthly | daily | explicit
    unit: shard                     # shard (default) | window — the dispatch unit
    time_field: delta_time          # per-observation timestamp column
                                    #   (a declared data_source column)
    epoch: "2018-01-01T00:00:00Z"   # dataset zero as an ISO-8601 UTC instant
    scale: gps                      # utc (default) | gps | tai
    units: seconds                  # seconds (default) | days
    windows:                        # explicit schedule only:
      - {label: melt-2019, start: "2019-06-01", end: "2019-09-01"}
      - {label: melt-2020, start: "2020-06-01", end: "2020-09-01"}
      - {label: scene-a, timestamp: "2021-03-14T12:00:31.024Z"}  # point form:
                                    #   the acquisition's OWN instant, copied
                                    #   from the item — not a rounded-off one

• Leaf naming is frozen: {full_id}_{window}.zarr, underscore separator, parse by splitting on the FIRST _. Generative labels are ISO-derived and hyphen-free (2025, 202511, 20251103), so lexicographic order = chronological order; explicit labels are opaque ([0-9A-Za-z-]{1,32}) and decode only through the declared list. quarterly is grammar-reserved but not implemented (validation rejects it).
• An explicit entry may be a point (issue #355): {label, timestamp} is sugar for the second-wide half-open [t, t + 1s), for time axes that are effectively discrete (single acquisitions, scene timestamps, isolated campaign instants). Each entry declares exactly one of timestamp or start+end — mixing them in one entry is rejected. The desugaring happens in zagg.config.get_windowing, so the manifest and every downstream consumer only ever see an ordinary {label, start, end} window. One second is the grammar's own resolution (boundaries are whole seconds throughout), which also keeps membership off float equality on observation timestamps (a sub-second timestamp normalizes to the whole second containing it). Consequence: two acquisitions within the same wall-clock second share a window, and two point entries inside one second are rejected as overlapping — as is a point that lands inside an already-declared range window, which is an ordinary overlap (see the validation paragraph below). If that ever bites, the fix is an explicit width key on the point form, not a guessed default; until it exists, any key on an entry beyond its own form's (label/timestamp or label/start/end; width included) is rejected rather than ignored.
• A point window that matches nothing fails SILENTLY — it covers only the whole second containing t, so the declared timestamp must be the acquisition's own instant, copied from the item, not a rounded-off approximation of it. STAC datetimes are rarely round seconds, so this is the likely first-contact mistake: timestamp: "2021-03-14T12:00:00Z" matches nothing for a scene whose datetime is 2021-03-14T12:00:31.024Z. On the point pipeline the window still dispatches and the worker's ge/lt filter matches nothing, leaving an empty leaf; on raster the group is dropped at dispatch (runner._raster_windowed_units) and no work unit, leaf, or warning is produced at all. Neither path errors.
• Boundaries are UTC calendar terms, half-open [start, end). Window bounds are converted to dataset units once at dispatch, using the declared epoch/scale/units and a fixed scale offset (GPS−UTC = 18 s, TAI−UTC = 37 s; stdlib datetime has no leap-second table) — boundaries are accurate to ≤ 1 leap second, none declared since 2017.
• Dispatch is one work unit per shard, emitting every window its granules span (unit: shard, the default — issue #586 phase 2). The ShardMap's per-granule time_start/time_end decide which granules belong to which window, and the worker reads the shard's granules ONCE, bins each read on time_field into the windows (the same half-open [start, end) predicate, applied to the read chunk instead of inside the read — an observation on a boundary instant belongs to the later window), then aggregates and finishes one window at a time — leaf, stamp, granule-id sibling, leaf column, refs, pointer — releasing each window's reads before the next one pools, so peak memory holds one window's slab beside the shard's reads rather than N. Under aggregation.streaming each window gets its own aggregator at the fan-out unit's own threshold, flushed on its own granule cadence and drained as soon as the read passes its last member granule (so the resident tail buffers are the open windows', not all N; under mode: merge every window's running state stays resident until its leaf is written), plus one shared cap on the open spill blocks together (SPILL_TMP_FRACTION of free /tmp), with every block close first joining all the windows' in-flight reduces so one block reduces at a time. A cap close is the one bulk-only fold, and on a config with no cross-block fold law it fails the shard (SpillOverflowError) where every fan-out unit would have stayed single-block: dispatch unit: window there. The leaves are byte-identical to the per-window fan-out's. unit: window keeps that fan-out — one invoke per (shard, window), the window's granule subset, an observation-level ge/lt filter pair injected into the read — for runs that want the smaller per-invoke footprint at the cost of reading each shard once per window. The shard event carries the shard's whole granule list plus each window's index list, so it is about N× a per-window event: at ~100 B per ATL03 s3 href the 250 KiB async (Event) budget holds ~2,400 granules, and a 4,600-granule pole shard × 7 windows is ~478 KB (every per-window event ~68 KB). Such a shard fails at dispatch on an async run (invocation="sync", or unit: window, is the remedy), and the per-leaf submap block is dropped — sub-maps deferred to the sweep CLI — at a correspondingly lower granule count. The unit is a dispatch choice: the leaves, the manifest's temporal block and the D19 semantic hash are the same either way. The raster path always dispatches per window (membership is per acquisition at dispatch) and rejects the key. Legacy shardmaps without granule times dispatch every granule to every window (the per-observation split keeps it correct) and need bounds.temporal to enumerate generative windows.
• Run records stay one row per leaf. A shard unit that emits N leaves writes N D20 records — each leaf's own sidecar, sub-map (its granule subset) and run-parquet row, with window naming the leaf, so the sweep's run-record discovery and Run.attach read them as before. The invoke-level telemetry (duration_s, duration_total_s — the billed wall, issue #589 — max_memory_mb, gb_seconds / est_cost_usd, the read phase, and n_obs_read — the shard's decoded rows, so the #374 read-vs-keep ratio is per invoke: n_obs_read over the rows' summed n_obs) repeats across the invoke's N rows and each carries unit_windows: N — the leaves the invoke emitted, windows the gate skipped not counted (null on a per-window unit) — so per-invoke quantiles stay the fleet-safety numbers and a per-invoke sum de-duplicates on (run_id, shard_key) where unit_windows is set. The stats rollup does exactly that (telemetry.merge: a shard node's rollup carries the bill of the one invoke that wrote its N leaves, and everything coarser sums shard rollups), as does tools/windowed_emit_measure.py; the dispatcher's own cost block and worker_pct_timeout read the invoke's result body, one per invoke. A window whose inputs are current under the skip-if-current gate is skipped inside the shard invoke (no record, as for a skipped unit); a window that kept no data reports the fan-out's benign no-data error and fails nothing. A window that fails costs only its own leaf — the others still land and keep their records and sidecars — but marks the invoke failed (error: "window {label}: ..."). Nothing re-fires it automatically (a body-level error is not retried and a failed status object is terminal): a re-run of the shard rewrites the failed window while the gate skips the windows that landed.
• The post-run tail reads leaves, not invokes. The shard above is a failed cell on every backend — its 500 and its failed status object on the fleet, cells_error on both — but the leaves that landed are in the store, and the run's tail treats them as the per-window fan-out would have: the sweep work set (the staged sweep's (node, window) units and the node close, which the rollup families ride where the store declares one — the families pass otherwise) is every result's stats records, each on its own success, never the invoke's status; a shard enters the root coverage.moc when at least one of its leaves stands (written by this run, or found current or refused by the gate), and the root time-range union takes those leaves' ranges alone. The failed window is in none of them. This is one rule on every dispatch path that runs a windowed config — the local backend and agg on Lambda over the sync, async and Event transports (runner._landed_coverage, sweep.leaves_from_stats_records) — so a backend never sweeps or covers what another would not. The zagg.client tail reads results through the same two readers, but today zagg.client and Run.attach refuse a windowed config, so a partly failed windowed run's tail is recovered with python -m zagg.sweep, not by attaching: the run parquet already carried one row per leaf, the failed window's unsuccessful, so its run-record discovery sees the same set. Only a worker that RETURNED reports its landed windows: an invoke killed outright (timeout, OOM) leaves no body, and on the fleet no D20 sidecar either, so the re-run rewrites those windows rather than skipping them.
• Stamps carry the truth, the manifest the schema (D15): each windowed leaf's commit stamp records its window label and the ACTUAL written time_range as ISO-8601 UTC strings (both ends at whole-second granularity, which narrows the tail by up to 1 s — see The commit stamp); the root coverage.moc summary carries the run's time-range union (cache, regenerable); temporal extent never lives in the manifest, which stays write-once. Appending a new year to a yearly store adds leaves the schedule already describes — no manifest touch; the explicit list is the noted exception (appending outside it re-templates).
• No per-cell morton array (issue #586 phase 3, specification §1.5 "The cell coordinate"): every window of a shard shares the shard's cells axis, so a windowed leaf does not store the coordinate — cell j carries the word of the shard's j-th child at the cell order (zagg.grids.morton.cell_words; one stride of 4,194,304 across an order-9 shard at cell order 19), derived from the leaf id and the rank. That is 32 KiB per occupied inner chunk and two objects per leaf not written — 8.4 MB per full order-9 shard, per window. Every write path drops it (the shard unit, unit: window, the streaming and spill modes, the raster writer, versioned leaves), on the local backend and the Lambda handler alike; unwindowed leaves keep their stored array unchanged. zagg's readers derive it when it is absent — the sweep fold and the column backfill pin the leaf's extent on a declared field instead, the tensor readers derive the words from the leaf's own stamp — and windowed leaves written before this change, which do store it, read as before. The leaf columns and overviews still store morton in every group. python -m zagg.pyramid_check holds the derivation against the data: its coordinates check requires every location word of a sampled cell to lie inside the derived cell, and reports a store with no located field as derivation only rather than as a pass.
• Coverage gains encoding: "full" (D14): a popcount at stamp time marks a fully-occupied subtree — no bitmap sidecar object is written, and readers short-circuit the exact intersection through the shard's own MOC membership. Partial shards keep the bitmap sidecar.

Validation: output.windowing requires the hive layout on a healpix grid; time_field must be a declared data_source column (the worker can only filter what it reads); explicit windows must be well-formed (frozen label grammar, start < end, unique labels, disjoint ranges — point entries are desugared first, so a point landing inside another window's range is a genuine overlap and is rejected). Range bounds render at whole-second granularity exactly as a point timestamp does — each bound truncates to the second containing it — so a fractional bound silently retimes that edge of the window: [12:00:00.0Z, 12:00:01.5Z) dispatches as [12:00:00, 12:00:01), dropping the declared half second of coverage at the tail. A range is never widened to recover the truncated fraction. The one case refused outright is total collapse: when both bounds render to the same second (say 12:00:01.0Z → 12:00:01.4Z) the window dispatches as an empty ge x/lt x pair, so it is rejected — the point form is the spelling for one-second intent. A sub-second range that straddles a second boundary (12:00:01.9Z → 12:00:02.1Z) is still valid; it renders to the one-second window its truncated bounds describe. The epoch is the one time value held to a stricter rule (issue #390): it too renders at whole-second granularity, but it defines every window conversion rather than one edge of one window, so a sub-second epoch is refused rather than truncated — 2018-01-01T00:00:00.5Z would shift every boundary by half a second, invisibly. Declare it at second precision. On the raster path (issue #247) membership is the acquisition's STAC datetime: time_field is optional (fixed to datetime) and the epoch/scale/units conversion knobs are rejected. Changing the windowing of an existing store fails the frozen-key manifest check like an orders change — clear the root first.

The manifest (morton_hive.json)

Written asynchronously at init (issue #252 hybrid): the local dispatcher writes it directly before dispatch; the Lambda leg fires the existing mode: "setup" hive branch as a fire-and-forget Event invoke immediately after the mode: "ping" preflight passes, so the manifest typically lands within seconds of init (best-effort: the Event invoke shares worker concurrency and runs retries-0, deferring to the finalize backstop under throttling or a dropped invoke) and a reader can start consuming completed leaves while the store builds. Finalize re-ensures it as an idempotent backstop (a frozen-key-matching manifest is accepted — no second PUT): worker Event invokes run with retries 0, so a lost async init write self-heals at end of run, and a run that crashes mid-fan-out still left a manifest at init. Otherwise never touched during a run (D6); the read-only frozen-key precheck (zagg.hive.validate_manifest) still runs before the fan-out so an incompatible existing store refuses up front on reruns (two concurrent first writes into a fresh root now collide within seconds of init, not at the losing run's finalize). That precheck is a read, so the same preflight also runs a write probe ([issue

495](englacial/zagg#495)); it is not hive-specific

(it runs on every s3:// ping, raster included), so it is documented with the other operator-facing credential material in docs/deployment/lambda.md.

With the manifest, every shard path is computable arithmetically with zero requests:

{
 "spec": "morton-hive/1",
 "dataset": {"short_name": "ATL03", "version": "007"},
 "semantic_hash": "9f1c…",
 "cell_order": 13,
 "shard_order": 9,
 "split_schedule": [1, 1, 1, 1, 1, 1, 1, 1, 1],
 "path_grouping": 1,
 "pyramid": {
  "spec": "zagg-pyramid/1",
  "overview": {"spacing": 2, "orders": [7, 5, 3, 1], "all_time": false,
               "fold_source": "cascade", "exact_levels": 1, "fields": {"…": "…"}}
 },
 "generated_at": "2026-07-10T12:00:00+00:00"
}

split_schedule is implicit under D2 (one digit per level down to the shard order) but recorded explicitly for forward compatibility. pyramid carries the overview family's order schedule and each field's D24 composability class — exact, approximate, packed (the §3 composition word, issue #515) or none (issue #201) — or, under zagg-pyramid/2, its grouped (node, cells) level schedule (Pyramid overviews, issue

382); an overview with an empty orders list is the declared-off form.

It is declaration only — overview zarrs are generated by a later post-process sweep (D11), never at fan-out time, and the sweep adds its materialized actuals under the same key. Declaring it is not birth-only: see Retrofitting the pyramid declaration.

fold_source declares how the sweep builds those levels (issue #376). Under the default cascade, only the finest exact_levels (default 1) fold from the source leaves; every coarser level folds the level below it, so the leaves are read once for the whole pyramid and a node's fold input is its 16 child overview slabs rather than its entire subtree. The cost is precision in the approximate and packed classes — a cascaded t-digest is a merge of merges, and a cascaded composition word re-quantizes once per level (specification §3.4) — which is why each level records the regime that made it, in the overview's own attrs and in materialized.fold_sources. The pre-#376 behavior is fold_source: leaves (every level re-folds the raw leaves, exactly); it is deprecated and kept only as an explicit opt-in:

output:
  pyramid:
    orders: [7, 5, 3, 1]
    exact_levels: 1        # cascade boundary; 1 is the default
    # fold_source: leaves  # deprecated: exact at every level, unbounded per-node input

Pyramid overviews (zagg-pyramid/2)

The declaration also comes in a second revision (issue #382; design record issue #381, as collapsed by the espg grammar ruling on the declaring PR): overviews — the leaf cell resolutions, and nothing else. A scalar is sugar for one resolution; a list is strictly descending and consecutive (the column tier — every ladder cell at or above the shard order — must step by one down to the shard order; [16, 13] is refused, naming missing [15, 14]), each member strictly between parent_order and child_order; omitted, the default is one resolution at the grid's resolved chunk order:

output:
  pyramid:
    overviews: [14, 13]   # leaf cell resolutions (consecutive); replaces orders/spacing WHOLESALE
    # overviews: 13       # scalar sugar for [13]

There is no above-shard configurability: everything coarser than the shard is the fixed every-order ladder — with d = coarsest leaf resolution − parent_order, every order from parent_order − 1 down to 0 carries one member at order + d. A /2 store is therefore inherently a multiresolution statistical grid: every HEALPix order from 0 through the base plus the native resolution, each level spec-guaranteed and individually addressable (pyramid: false is the single-resolution opt-out; skipping levels when reading is a reader-side choice, never a store property). Spelling overviews writes the manifest block as zagg-pyramid/2 with the fully expanded (node, cells) list at block level — pyramid.overviews, the store-wide product declaration; readers never re-derive the ladder — while the singular pyramid.overview family dict keeps the sweep leg's execution regime (all_time, the #376 fold keys, fields, materialized) exactly as under /1. Since the issue #384 default flip, omitting the knob also declares /2 — at the grid's resolved chunk order — whenever that order is strictly interior to the shard's resolution window; raster configs, explicit legacy orders/spacing schedules, and K == 1 grids keep /1. The grammar's validation rules and the ladder law are normative in the specification §4.4–§4.5; refusals are loud and named, never silently widened. Two practice points:

• Declaring is free. A declared-but-unswept level costs nothing (declared intent and swept actuals are separate — #381 point (11)), and the fixed ladder makes every coarser level spec-guaranteed without spelling anything. Sweeping, not declaring, is the operational decision — one aggregation template serves both a state-scale AOI (sweep immediately) and disjoint small deployments (sweep later, or never).
• A /2 store is swept by the staged sweep (issue #384) — the /1 overview family generates nothing for it (its per-level leaf fold is exactly what the column regime ends) and reports the routing in its counts (regime: "stages"). /1 stores sweep exactly as before — the #379 cascade remains the path for pre-column stores, and stays the regime for finer-than-base appends (where gen 3 lives).

A windowed store (Time windows) additionally declares spec: "morton-hive/2" and a temporal block — schedule, time_field, epoch/scale/units/calendar, the explicit windows list, and the append policy. Temporal extent is deliberately not manifest data (D15): actual ranges live on leaf stamps (truth) and the root summary (cache).

A rerun into an existing root verifies the manifest's frozen keys match the run's own configuration (spec, dataset, cell_order, shard_order, split_schedule, temporal) and fails loudly on a mismatch — the hive analogue of the flat layout's shard-map signature guard. The sweep-mutable pyramid block and the generated_at timestamp are deliberately excluded, so a swept store still resumes (and the sweep's pyramid declaration is preserved, not clobbered).

Re-templating does not remove existing leaves. Overwriting the manifest replaces only the JSON — committed leaves written under the old configuration would survive, stamped and walker-discoverable, and (because mixed shard orders are legal under D2) indistinguishable from intentional data. The writer therefore refuses an overwrite that changes the frozen keys while the digit tree has any {sign+base} children (one delimiter-LIST): clear the store root, or pick a new one, before writing with a different configuration.

The templates declare the ladder (issue #620)

The three packaged hive templates ship the /2 ladder rather than pyramid: false (espg's ruling on PR #622):

template	output.pyramid.overviews	cells / shard	column tier
atl03_tdigest_strata_healpix	[16, 15, 14, 13] — the v4 store's	19 / 9	16 … 9
atl03_tdigest_healpix_hive	[16, 15, 14, 13] — follows the strata one	19 / 9	16 … 9
gedi01b_waveform_healpix_hive	[16, 15, 14, 13] — the same relation to its shard order	18 / 9	16 … 9

The coarsest member is the shard order + 4 in each, so the fixed every-order ladder above the shard bottoms out at node 0 / cells 4 — the same 13..4 dense ladder the v3 ATL03 store declared as {overviews: 13}, plus the finer leaf members. GEDI's cells are order 18, so its finest member sits two orders above the base data where ATL03's sits three; the tier is otherwise identical. The declaration is hash-invisible (D11), so declaring it moved no template's semantic_hash (tests/test_live_store_templates.py pins the digests and the ladder).

The ATL03 templates are windowed yearly from birth (issue #638, espg's ruling 2026-10-12). atl03_tdigest_strata_healpix and atl03_tdigest_healpix_hive carry an output.windowing block — schedule: yearly, time_field: delta_time (declared under data_source.variables), the 2018-01-01 GPS epoch in seconds — and spell neither all_time nor icechunk: a windowed config resolves pyramid.all_time to true (the all row is allocated at birth, issue #623) and a windowed hive store gets its Icechunk repo at birth, one manifest per row (issue #604). So a run from either template builds the v4 shape — per-year leaves, the all row, the ladder, the repo — with no flags (tests/test_template_windowed_build.py builds one from each on the local backend and runs pyramid_check over it). The templates also spell output.sweep: "stages" although the default resolves to it, because the Icechunk commit regime keys on the explicit key (zagg.icechunk_refs.ladder_walks, PR #622 ruling (1)): spelled, a run from a template commits refs up the ladder — one commit per commit-order node, the regime that scales to the globe (spec §11.4) — where an unspelled one would commit per leaf (espg ruling on PR #639). A windowed store is the one that scales (windowed emit cut worker peak memory 3.7 → 1.56 GB, issue #586; a window invoke reads only its year's granules), and the schedule is fixed at birth and only folds up later, so the template carries the schedule the product is meant to have; monthly is a deliberate per-store choice (≈6.5× the compute). Deleting the block is the opt-out — an unwindowed build stays legal. Unlike the ladder, windowing is hashed (a D13 leaf-shaping key: a windowed store is a different product), so the ATL03 template's digest moved and the pins say so. The same templates set the pyramid digests' overview_delta to 128 centroids (the v4 ruling: the stored 15/16 columns at ~12 GB rather than ~95 GB at 512, the viewer's order-13 fill 1.29 GB → ~0.32 GB; the leaf delta: 4096 is unchanged, and the knob is hash-invisible). GEDI stays unwindowed and at 512 for now: its delta_time is a shot-rate column (declared on the shots level and broadcast to samples), and window membership is decided on a base-rate data_source.variables column only, so the block is refused by name there; its overview budget is issue #640. Two consequences follow for an operator:

• A run from any of these templates chains the staged sweep by default. output.sweep is unset in all three, and an unset sweep on a hive config that declares a ladder resolves to "stages" (zagg.config.get_sweep_mode, the staged sweep), so the ladder — and the rollup families riding it — folds in the run tail, scoped to the run's own footprint. sweep: true keeps the families pass alone; false keeps neither.
• A store templated before this may carry a different declaration, and its manifest is the store's truth. The older templates built pyramid: false stores, and the v3 ATL03 store declares {overviews: 13}. pyramid.spec is written once, at init; an append from the new template never rewrites it, and the staged sweep refuses a /1 manifest (see the Migration notes on PR #622). Since issue #635 that refusal costs nothing: the fleet dispatcher reads the store's declaration once before it fans out, so the run no-ops with one warning naming --declare-pyramid instead of billing one dying invoke per dispatch node. sweep: true is still not the opt-out, and the reason is now only the leaf side: the stage fan-out already stops itself, but the leaf side follows the config's ladder and not the manifest. The leaf worker writes [16..9] columns (leaf_column_plan) that the store declares nowhere. The Icechunk leaf seam stages refs for the config's leaf levels (leaf_level_cells) into a repo whose groups were built from the manifest. The 16 group does not exist there, so every new leaf's commit fails, its base refs included, and the leaf drops out of the index (fail-open: icechunk.error in its sidecar). Append to an older store with output.pyramid set to what its manifest declares: false, or {overviews: 13} for the v3 store. That config writes no undeclared column and indexes only the store's own levels. A false declaration also resolves the default sweep to the families pass. Or retrofit the store first. The retrofit is --declare-pyramid with the config the store was built from; declare_pyramid replaces a differing declaration, so the template as shipped re-declares an older store's ladder as [16, 15, 14, 13] — intend that, or hand it the store's run-record config (pyramid_upgrade.md).

Leaf columns (zagg-column/1)

Under a /2 declaration the leaf worker itself writes the finest pyramid levels, at aggregation time, while the shard's cell data is resident (issue #383; the fleet side of #381 points (1)–(3)). The gate is the writing run's own config, not a store read — workers never open the manifest, and the manifest's pyramid block (excluded from the frozen keys above) MAY lag a config-only change — so a /2 manifest does not by itself imply columns on disk: read the columns, not the block. Each (leaf, window) gains one column artifact beside the leaf under its own node directory — {window}.pyramid.zarr, all.pyramid.zarr unwindowed — holding one resolution group per within-footprint level: the declared base resolution(s), every ladder rung down to the shard order, and the node-order member (one cell — the leaf's whole-footprint aggregate, its universal partial for every coarser cell, which is why no coarse level declared later ever rewrites a leaf). Groups fold from the raw resident cells only (merges-from-raw 1; exact classes exactly, digests by the order-independent k-way merge), so a column group is byte-identical to the sweep kernels' fold of the committed leaf. The write discipline is the leaf's own: template → groups → role: column + zagg_column attrs → one commit stamp last, then the {stem}.stats.json sidecar — an unstamped column is debris. A run that declares no leaf-node levels (or no composable field) writes no column and deletes any the previous declaration left at that (leaf, window), so a column never outlives the declaration that wrote it — which is why an absent column is not by itself a fault: under a declaration known to carry leaf columns it is a torn worker, and the repair is re-invoking the idempotent leaf, never a sweep-side fold from raw cells.

Layout (issue #631, zagg.column_shards). Every column array is one object per leaf, and its chunking is a function of the group's own (node, resolution): a group of fewer than 4^5 cells is one whole unsharded chunk — every column written before #631, and the whole default tier {13, 12, 11, 10, 9} — while a group of 4^5 cells or more (orders 14–16 under the v4 ATL03 rebuild's overviews: [16, 15, 14, 13]) is one sharding_indexed object on inner chunks of min(4^5, cells / 4) cells: 16 inner chunks of 1,024 at order 16, four of 1,024 at 15, four of 256 at 14. The inner chunk is ~0.5 MB of saturated signal digest at the 128-centroid cap, is a power of four (a viewer's window block never straddles one), and never exceeds the staged sweep's fold block, so a stage gather over a fine column reads at most one inner chunk's worth of cells per block read through the shard index — one 1,024-cell inner chunk above 4^5 cells, the whole four-inner-chunk object at exactly 4^5. The Icechunk companion indexes a fine column level at its inner chunk; a companion repo initialized before #631 still chunks those levels whole, and every open that vets the geometry — the per-run init, the leaf vet, the ladder and declare_pyramid — refuses it until it is cleared and re-initialized (the finalize, mutate and history opens do not vet the level geometry), the /1 → /2 backfill re-lays a whole-chunk fine column as structure drift, and pyramid_check holds every column array to the rule. Byte grammar: specification.md §4.6.

Retrofitting the pyramid declaration

The pyramid block is normally written at template time (build_pyramid_block, issue #201), but declaration is not birth-only: a store templated without it — any pre-#344 store — can gain (or change) the declaration later, because the block is a pure function of the pipeline config plus the manifest's own shard_order, and pyramid is already the one manifest key legally rewritten after templating (D11). The retrofit is zagg.sweep_overview.declare_pyramid(store_root, config), or from the shell:

python -m zagg.sweep s3://bucket/store --declare-pyramid config.yaml

(declaration-only: no sweep pass runs in the same invocation). The tool derives the block through the same code path template time uses, then validates it against store truth before writing, in two layers:

• Semantics — the config's D19 semantic_hash must equal the manifest's frozen one, so a config that did not build this store cannot install its fold laws. This is the layer that covers reducers: no leaf records which function produced a field, so nothing downstream could catch a config declaring max over a store of minima. Adding output.pyramid to the original config hashes identically, so the retrofit never false-refuses — the whole pyramid block is deliberately outside the semantic core, and keeping this workflow working is one of the reasons why (the D19 hash epoch of issue #415 put the leaf-shaping output knobs into the core, listed in zagg.semantics.OUTPUT_LEAF_SHAPING_KEYS; pyramid is not one of them). A pre-#299 manifest carries no hash; the comparison is then skipped, loudly.
• Typing — every declared field must exist in a committed leaf with the declared dtype (dense fields), or the declared ragged element dtype and inner_shape at a ragged spec revision this zagg understands (t-digests).

Drift in either layer refuses loudly, naming the field and the mismatch, and nothing is half-written. What is not cross-checked: the config's dataset identity and its grid parent_order/child_order against the manifest's orders — the declared overview orders are checked only for being ancestor orders of the store's own shard_order. A store with no committed leaf yet is still declarable; the field checks are then skipped, loudly, and the summary says which case it was. The rewrite is idempotent (an identical declaration is not re-PUT) and preserves any materialized actuals the sweep has recorded. output.pyramid: false installs the declared-off block — recording absence is a valid retrofit, so a reader can skip probing for overviews entirely (the D24 option-A posture the block enables; no reader in this repo consumes the declaration yet), and overviews at now-undeclared orders are left in place as regenerable-cache debris (D24). After a retrofit, the overview family materializes the declared orders on the next sweep — it is in DEFAULT_FAMILIES, so a plain python -m zagg.sweep <root> picks it up (the fold itself is issue #201 / PR #344; the retrofit is issue #358). On a store with an Icechunk companion repo the same step then commits the declaration into the repo (zagg.icechunk_ops.declare_pyramid, spec §11.4 Operations: new level groups, the multiscales root attrs; fail-open, reported under the summary's icechunk key), so both planes are declared in one step.

A config that spells overviews (Pyramid overviews) retrofits the zagg-pyramid/2 declaration through the same path: the level entries are re-validated against the manifest's own shard_order/cell_order (the store's truth wins over the config's grid block), and any /1-era materialized actuals are preserved verbatim across the revision bump. The staged sweep (below) then materializes the ladder from the leaf columns; a retrofit onto a store whose leaves predate the column regime under-covers loudly (source_children) until the leaves are re-run — declared-but-unswept stays a legal recorded state, not an error, and the #379 cascade remains the /1 path for pre-column stores.

Partitioning a sweep that will not fit

A sweep pass holds the fold state for the subtree it is walking, so on a large store a single pass can exceed the worker's (or the laptop's) memory. Split it into 2^n disjoint morton-subtree partitions:

python -m zagg.sweep s3://bucket/store --partitions 16

Each partition folds one order-k subtree per HEALPix base cell (n = 2k), so peak memory is bounded by the partition rather than by the store, and the command prints one summary per non-empty partition. --partitions must be a power of four — a morton digit is 2 bits, so that is where the split lands cleanly; an odd 2^n is refused with the two valid neighbours named.

Partitions are disjoint by construction, so they are also safe to run concurrently: no two write the same object, and each is independently idempotent (a re-run folds only what actually changed). What a partitioned pass deliberately does not do is anything above the split order — the coarse rollup levels, the root coverage.moc refresh, and the manifest's pyramid.materialized bookkeeping all span partitions and are left to a coarse-level finisher (issue #377).

The run tail partitions this itself on the Lambda backends ([issue

610](englacial/zagg#610)) — for a run that walks

no /2 ladder, the /1 retrofit regime; any other /2 run fires no families pass at all and its families ride the staged sweep instead (see "The rollup families ride the same walk", below): the end-of-run families pass is split 4^k ways from the run's own leaf count — k from ceil(log4(leaves / 128)), then refined until no partition holds more than 128 leaves, because a regional store's leaves cluster (the 2,959-leaf California store's 64-way split held 1,050 in one) — fired one mode="sweep" invoke per non-empty partition, awaited on the copy of each one's record it drops as families-p<i>of<n>.json under the run's status prefix (<store>.status/run-<run_id>/; the sweep_stats_<ts>_p<i>of<n>.json at the store root still lands), and then finished by a partition-less invoke awaited on its own families-finisher.json; a run of at most 128 leaves is the one pass it always was. The outcome rides the run summary and the zagg.client handle as families_sweep: {partitions, fired, landed, finisher, accumulators, run_id, records_from, duration_s}, with finisher "ok", "records_short" (a partition's record never landed), "timed_out" (the finisher's did not) or "dispatch_failed" (an invoke raised after others fired, or the pass was fired and never dispatched at all — fired: 0; error says why), so a short root temporal section is reported rather than discovered later. The staged leg reports the same way — stage_sweep, the dispatcher's whole summary, barrier_timed_out and short_orders included — and tail_sweep_short is the verdict over both: one reason per shortfall, [] when the tail left nothing short, logged as one warning and printed by python -m zagg as TAIL SHORT: lines. The finisher reads no leaf for the stats/moc/submap families (the overview family's own fold on a /1 store is unchanged and still opens its leaves; a /2 store refuses it first): each partition's record carries, under families.moc.accumulator, the §10 contributions it read — every visited shard's envelope word and counted cover per window leaf (the §10.3 block grammar), the uncounted set, the declared fields and the route tally — and when every record landed the finisher event names them (finisher: {of, records_from, accumulators}; accumulators in the outcome is how many), the worker folds the blocks, and its walk starts from the split-order rollups the partitions wrote, so the coarse levels, the root coverage.moc and coverage.toc compose from rollup reads and n record GETs, byte-identical to a single pass. A missing or unusable record, or a work set holding a shard no partition visited, sends it back to the leaf walk, which its own record says (finisher.fallback); an event too large to carry its leaves inline (discover: true) carries no finisher block at all, since the discovered work set is the store's, not this run's. The local backend sweeps in-process and needs none of this.

On the Lambda backends the tail fires that finisher itself. For hand --partitions passes (the local CLI, or a tail whose finisher did not land), until a finisher lands, the coarse levels of the JSON rollup families (stats, moc, submap) can be picked up by following a partitioned sweep with a plain python -m zagg.sweep <root> --families stats,moc,submap: the partitions' work is skip-if-current, and an interior fold reads its children's rollups rather than the leaves, so the extra pass is cheap. The overview family's deferred coarse levels likewise fold from the finer overviews the partitions already materialized (the default fold_source: cascade, issue #376) — but that follow-up is not cheap the way the JSON one is: an overview's currency check folds before it compares, so every leaves-fold level re-reads its raw leaf zarrs store-wide even where nothing changed. And the follow-up performs only half the deferred manifest bookkeeping: its RMW unions the materialized orders, but records a fold regime only for a level that wrote in that same pass — the regimes the partitions wrote under ride each partition's own run record as materialized_fold_sources, and reach the manifest only with a finisher that carries them. Leaves-fold levels are not only the deprecated fold_source: leaves, where that is every level — the blow-up the cascade exists to remove: under cascade the finest exact_levels levels fold from the leaves too, as does any level declared a wider gap than the node slab can divide. Prefer the finisher for the overview family. (A zagg-pyramid/2 store never gets that far: its overview materialization is the staged sweep's, below — which retires this two-call recipe wholesale for /2 stores, since the stage run carries its own designated finisher.)

The staged sweep (issue #384)

A zagg-pyramid/2 store's above-shard ladder is materialized by stage workers over the leaf columns — a raw leaf is never read above the shard:

python -m zagg.sweep s3://bucket/store --stages --scope 3111,3112   # CLI backstop
python -m zagg.sweep s3://bucket/store --stages --scope all --partitions 16
python -m zagg.sweep s3://bucket/store --stages --scope all --pipeline-run-id <run id>

--scope is required under --stages ([issue

620](englacial/zagg#620)): this CLI is the recovery

form, so it names the subtree being recovered, and the whole store is asked for by name (--scope all). An unscoped pass re-visits every node, and a node that is already current still costs its visit.

or in code zagg.sweep_stages.run_stage_sweep(root, leaves, scope=...), or chained immediately after a fleet run by output.sweep: "stages" (auto-scoped to the run's own footprint) — from python -m zagg and from the client facade's Run.dispatch alike ([issue

588](englacial/zagg#588); the facade's post-run

tail chains it after the run record and the rollup sweep, and the summary rides the handle as handle.stage_sweep). That chaining is the default for a hive store that declares a ladder since [issue

620](englacial/zagg#620) — the sweep belongs to

the append, not to an operator call; sweep: true (or "families") keeps the families pass alone and false (or "none") keeps neither. That default reads the config, so the fleet dispatcher checks the store before it fans out: one manifest GET, and a store that declares no /2 ladder fires nothing at all ([issue

635](englacial/zagg#635) — the dispatcher's only

read, with D8 amended for it; the gate is the workers' own ladder_entries, so a /2 store's run is unchanged). Work is discovered from the run records (listing-based; the root coverage.moc is an accelerator for sibling candidates, never the source of truth — a fleet append with no subsequent sweep leaves it stale, and discovery still finds the new leaves).

The same sweep also runs entirely worker-side, as a fan-out of Lambda stage workers with the dispatcher invoking and polling and never writing (issue #519) — which is the only way to build the ladder on a store whose bucket policy names the fleet execution role as the write identity. The wire grammar, the sequencing and the permissions are in docs/deployment/lambda.md. When the process that launched a run dies before or during its chained sweep, the operator's steps are in Recovering a run whose launcher died.

Cadence. Ladder orders are grouped into dispatch tuples of tuple_width consecutive orders (default 3: [8,7,6] → [5,4,3] → [2,1,0] on an o9 store). A stage worker folds its tuple's levels finest first: a gather level (cells at or above the shard order) concatenates its children's column members, a merge level cascades its four children's artifacts at the level one order finer — the ones the same worker has just folded, from memory, or the previous tuple's, read back off the store (issue #620). A merge node therefore reads four slabs whatever its subtree holds (1,024 cells in, 256 out at the reference geometry), where it used to re-read its whole footprint at the leaf columns' relay resolution (18.9M cells at order 0, past the 900 s wall at the dense order-1 node). The worker's own column carries only the members a coarser gather consumes — nothing for the merges, so a dispatch node below which nothing gathers writes none. The values are a fixed function of the ladder, so the cadence changes no values: --tuple-width 1 and --tuple-width 3 build the same ladder, and merges_from_raw records the fold depth (at a gather, the depth of the column groups it concatenates; one more than its children's at a merge).

The run tail sizes the cadence itself ([issue

610](englacial/zagg#610)). A dispatch node folds

its whole subtree down to its tuple's child order inside one invoke, so a fixed width puts 1 + 4 + … + 4^(width-1) nodes on one worker wherever the store is dense — 21 at width 3, which is what walled the v3 ladder: the [2,1,0] tuple's base-cell-3 invoke hit the 900 s wall after 12 of its 16 order-2 nodes, no record landed, and the barrier was waited out. Capping the nodes an invoke is handed (max_nodes_per_invoke, already 1) does not help: the subtree is one node's work. (The two compose rather than substitute — n nodes on one invoke fold up to n × a node's own subtree — so the schedule is sized against the target shared out per node.) So the Lambda tail asks for a sized schedule — tuples laid from shard_order − 1 downward, each the widest width whose fattest dispatch node folds at most STAGE_TARGET_NODES (8) nodes, with the remainder at order 0 (issue #637). The tail hands the dispatcher no coverage MOC (a dispatcher may not read one itself, D8), so the fold is the dense bound (4^width − 1) / 3 — 21 at width 3, 5 at width 2 — which never under-estimates; a caller that does hand a coverage MOC in gets the fold measured over the set the worker folds (the run's leaves ∪ that coverage), and may keep the full width where the store is genuinely sparse. The sizing reads nothing from the store either way; the dispatcher's one read is the ladder gate's single manifest GET (issue #635, below). A dense o9 ladder's three width-3 tuples folding 21 nodes an invoke become five tuples of uniform width 2 — 7:[8,7] 5:[6,5] 3:[4,3] 1:[2,1], then the one-order remainder 0:[0] — folding 5. Alignment matters more than width: on the California o9 shardmap (731, 205, 64, 23, 10, 6, 2, 1, 1 interior nodes at orders 8..0) that is 236 tail invokes, where the earlier rule — subdividing each fixed width-3 tuple inside its own span, so the widths alternated 2/1 across six tuples — cost 287, and a bottom-aligned width 2 (stage_tuples(9, tuple_width=2), remainder 8:[8] on 731 nodes) would cost 808. A store sparse enough to carry the full width keeps it, top-aligned, so a sized schedule equals the fixed-width one exactly when shard_order is a multiple of the width. Because a subtree's node count steps by powers of four, any target in [5, 20] picks width 2, [21, 84] width 3, under 5 width 1. The default Icechunk commit_order is still the fixed schedule's finest dispatch order (6 at o9, zagg.icechunk_refs.finest_dispatch_order), and the ladder commit is made by the sized tuple whose span contains it — 5:[6,5], so by the order-5 dispatch nodes, each committing up to four order-6 base manifests (four when dense) in the same invoke that folds its tuple. The per-invoke fold target counts ladder nodes only, not commits, so under commit: "ladder" a committing invoke carries up to four base manifests' refs beyond the folds it was sized for (one before, when the commit tuple dispatched at order 6). Such a tuple dispatches at an order no single width lands on, so its events carry their span (child_order) outright; a worker deployed before this refuses one by name rather than folding the wrong span. The cadence is still a dispatch knob like --tuple-width itself — a sized build's ladder overviews are the fixed-width build's, which is what the oracle pins. Not every byte: the group attrs carry per-run provenance that is grouping-dependent by design (source_children, the summed child generation), and a sized build may write stage columns a wider grouping never needs. python -m zagg.sweep --stages is unchanged (the CLI has no 900 s wall).

A short tuple does not stamp the manifest. The finisher records per-level actuals from the run's stage records; a tuple whose unit record never landed has none, and before issue #610 the levels it owned were stamped from whatever the run did observe, exactly like the complete ones — the v3 run wrote actuals complete for orders 8..3 and short for 2..0 with nothing to tell them apart. The dispatcher now names the short orders to the finisher (short_orders, from the tuples it is missing a unit record for), and their level entries are left as they stood while the orders that landed are recorded as usual. A tuple is short on either fan-out, so a lost all-time close record withholds that tuple's window levels too, even where their per-window records all landed — the safe direction: a withheld level keeps the actuals it had until a pass observes it in full. The finisher's record says which it withheld (short_orders, finisher.actuals_withheld), the tail warns, and the next staged pass records them.

Units. The unit of stage work is the dispatch node on an unwindowed store and the (node, window) pair on a windowed one (issue #586): each window's overviews and stage column come from that window's leaf columns alone, so a node's windows are independent. The fleet runs them concurrently, one invoke per window per node; the CLI and the local backend run the same units serially. A node's window units share no object — each writes only its own {window}.zarr / {window}.pyramid.zarr and reads its skip key off the artifact itself — so one that fails costs its own window and nothing else, and is named in the stage record (unit_errors). Each node then gets one close (output.pyramid.all_time, which a windowed config declares — resolved to true when omitted, refused when false, [issue

610](englacial/zagg#610) phase 4): the all-time

fold, a k-way merge of the node's per-window overviews into all.zarr (overview-sized objects at the same node, never leaves — its cost follows the window count, not the subtree). It covers every window the node has, not only the ones the run touched, records the windows it folded (source_windows), and is one more merge from raw than its sources — 2 at a gather level, 3 at a merge level (specification §4.4). A store written before the fold was required, whose manifest does not declare it, has no close and pays no unit for it. The declaration is the store's: the fleet dispatcher reads it back from the first window unit's stage record (closes, taken from the manifest the worker ran against), so a run whose config omits or contradicts all_time still closes exactly the nodes the CLI would.

The rollup families ride the same walk ([issue

610](englacial/zagg#610) phase 4). The

stats / moc / submap rollups and the §10 per-shard TOC contribution are the same bottom-up walk the ladder does — a (node, window) unit already reads the leaves under its node — so they no longer need a fan-out of their own. Named with --families on --stages (or families= on run_stage_sweep / run_stage_sweep_fleet) — and by every run tail that walks the /2 ladder: runner._cascade_families, the same config-only gate as the Icechunk commit pin (output.sweep: "stages" plus a /2 declaration with a composable field), hands CASCADE_FAMILIES to the staged sweep on the local backend, agg on Lambda and the client facade alike, and the tail then fires no families pass; a run that walks none (a /1 store, or "stages" on a config that declares no ladder) keeps the separate pass exactly as before — they fold in each dispatch node's close, over that tuple's own span: the finest tuple's span starts at the leaves (the stats sidecar, the D4 stamp, the leaf sub-map, the §10 temporal.toc record — all through the invoke's one store handle), and every coarser tuple's starts from the rollups the tuple below it wrote. The span decomposition is exact, so the rollups are byte-identical to a whole-tree python -m zagg.sweep pass at any tuple_width. The close is where they ride because a rollup object merges every window of its node (its windows key): the node's window units run concurrently and would read-modify-write one object. A windowed store without the all-time fold would have nowhere to put them, which is why a windowed config must declare it — pyramid.all_time resolves to true when omitted and is refused when false (espg's ruling on [PR

623](github.com/englacial/zagg/pull/623#issuecomment-6116823949),

issue #610 phase 4) — and the fleet dispatcher refuses a windowed caller that says otherwise before anything fires. Its all_time is still only the caller's first guess about the store: a manifest that withdrew the fold is reported (families_unswept) from the store's own answer, read back off the first landed window record, and those rollups are then swept by hand. That is the case for every append to a windowed store written before the fold was required (its manifest says all_time: false, and ensure_manifest never rewrites the block): the dispatcher fires no close, warns that the rollups went unswept, and the fix is the one-time python -m zagg.sweep <store> --declare-pyramid <config.yaml> retrofit (or the hand --families stats,moc,submap pass), per the recovery runbook in Recovering a run whose launcher died. The run's finisher then composes the families' store-root singletons once — the §10 temporal section inside coverage.moc and its coverage.toc sibling — from the base-node rollups (≤ 1 GET per base cell) plus the accumulator blocks the stage records carry, reading no leaf. Where the moc family rode and found base-node rollups to compose from, its own finish owns the root coverage.moc refresh and the finisher's step 1 stands down (root_moc_from: "families"); otherwise step 1 runs from the work set as it always has, because only that family writes the object. Which families rode is taken from the records, like closes, never from the finisher's event. On the 64-leaf local fixture the reads are the same 192 leaf reads either way: 192 in the one families invoke before, ≤ 12 per close unit across 16 units after, and two fewer end-of-run invokes.

Memory. A stage worker folds a level one block of output cells at a time — read the child members covering the block, fold, write, drop — and never holds a level whole (zagg.sweep_fold). A block is 4^5 cells; a stage-column group wider than that is laid on inner chunks of exactly one block, so the block written and the block a parent reads back are single chunk objects. Nothing is spilled to /tmp: the inputs already live in the store. The stage rows report fold_blocks, fold_cells_read and fold_peak_cells (the most source cells any one block held).

Which run a sweep completes. A sweep has its own run_id. When it is chained after an aggregation run, that run's id rides along as pipeline_run_id and is written into the sweep's run record (sweep_stats_{ts}_stages.json) and every stage record (issue #593), so the record names the run it vouches for instead of being linked to it by time. A standalone --stages pass names the run it is completing with --pipeline-run-id; without it the key is null and the pass vouches for no run.

Scope. The only argument a sweep takes about where is an optional node-prefix set — a MOC; a shardmap is accepted as sugar (its keys are the prefixes). Scope selects which dispatch nodes are invoked; a dispatched worker folds all children on disk, so an update adjacent to prior data folds the old neighbors in automatically, and clean nodes no-op under the generation ratchet. --partitions composes with scope by intersection, swept under one admission.

Concurrency (the ruled matrix):

regime	allowed?	governed by
fleet ∥ fleet	yes, iff their (window, shard) write sets are disjoint	the existing leaf single-writer law
fleet ∥ sweep	yes	disjoint object sets; the stage worker validates every column stamp before and after reading its groups and re-reads on movement; once a column has served a block, a rewrite under it fails the artifact being folded, which is folded once more from fresh reads, so a mid-read leaf rewrite never feeds a torn column into a merge; a mid-sweep append is recorded-and-healed under-coverage
sweep ∥ sweep	no — serialized per store	the admission lease

Only pyramid sweeps serialize. Admission is one conditional PUT of the store-root intent object sweep.lease.json (spec §4.8): a live intent refuses naming the running sweep; an expired heartbeat is claimable, and the claimant simply completes the partial prior run. The lease is control plane — no data object is ever locked; it is what makes "every data object has exactly one writer, ever" true across runs. It is store-granular by correctness: scope-disjoint sweeps still converge on shared coarse ancestors (base cells, the root MOC, the manifest). The designated finisher-worker (never the 12 base cells) owns the root singletons after the root tuple — the root coverage.moc refresh, the manifest RMW writing the per-entry actuals, the aggregation.yaml lifecycle touch — and deletes the intent as its final act. A run that dies mid-sweep leaves its intent to expire into claimability; stage stamps carry the run id, and a worker that sees a foreign fresh stamp aborts loudly (the residual-race backstop).

Soft barriers. Stage order is a scheduling preference, not correctness: a tuple run before its finer tuple landed under-covers loudly (source_children) and self-heals on the next pass — the skip gate keys on summed child generations, so a healed child forces the parent rewrite. That key is a triple — leaf count, newest child stamp, and the set of run_ids those stamps carry (issue #417, specification §4.5) — because stamps resolve to one second: without the run ids a child rewritten inside its own recorded second at an unchanged leaf count would read as current and the parent would keep the stale fold.

Aging. Stage reruns refresh the ancestor artifacts they revisit (a rewrite is a fresh PUT), and the finisher's manifest RMW + root-MOC write refresh the store-root objects every run — so a store swept on any cadence keeps its pyramid's LastModified moving. A store that only ever SKIPS (all-current fleet runs, no sweeps) still ages its sweep-written ancestor overviews out under a bucket lifecycle rule: the per-unit lifecycle touch covers leaf footprints, not ancestor artifacts (the PR #397 finding — this is the recorded posture, not an oversight; re-sweeping is the refresh).

Checking a swept store (python -m zagg.pyramid_check)

The acceptance check for a swept pyramid (issue #434) is read-only and runs from any machine that can read the store:

python -m zagg.pyramid_check s3://bucket/product.zarr            # your credentials
python -m zagg.pyramid_check s3://public-bucket/product.zarr --anon

It prints one line per check and exits nonzero on any FAIL. It derives what it expects from the leaves, the artifacts' own recorded attrs and the geometry — never by calling the fold it is checking — and it samples: the value checks read --sample-nodes nodes per ladder order and --sample-cells cells per node (--full, for a local fixture-scale store only, reads everything and can re-run the sweep to prove it a no-op).

On an unwindowed store the checks are declaration (the manifest grammar), coordinates (leaf cell words against the located fields), materialization (a committed all.zarr at every declared ladder node), columns (a committed leaf column per leaf), readback (the attrs each artifact records, held to specification §4.4/§4.6), and counts / digests / composition (each sampled cell re-folded from the leaf columns, and each column from its leaf).

A windowed store (issue #586) is one ladder per window, so the same checks run per window over that window's objects, and every finding names the node and the window (-511[2019], window 2019: …):

• records — where the manifest declares a per-centroid temporal field (§8.3): each sampled (leaf, window) is read as the families sweep reads it, record first. A leaf with a usable temporal.toc passes, a leaf holding no clocked observation needs none, and a stamped leaf with clocked observations but no usable record — which the sweep would count as coverage only, under uncounted_shards — fails by leaf and window. With no temporal field declared the check is SKIP.
• materialization — a committed {window}.zarr at every ladder node that has a leaf of that window beneath it; columns — a committed {window}.pyramid.zarr beside every (leaf, window). Every check derives from the leaf roster, so the roster is cross-checked against the store: a window with an overview at an order-0 node the roster does not reach (every window's ladder runs to order 0), and, under the run-record roster, a shard the root coverage.moc covers and the records lack, both FAIL by name — rerun with --roster list. A recorded shard missing from the MOC is the MOC lagging (its write is fail-open) and is only named.
• readback — each artifact's window key must be the window it is filed under, and only the all-time fold may carry source_windows (both checked on every artifact, not a sample — the all-time folds' window: all too, reported under all_time), plus the per-window regime, merges_from_raw and source_children of §4.4 on the sampled nodes.
• counts / digests / composition — a window's gather levels re-folded from that window's leaf columns, its merge levels from their children's artifacts in that window, and its columns from that window's leaves, for --sample-windows windows (default 3). An overview built from another window's leaves fails here.
• all_time — where the manifest declares pyramid.overview.all_time: a committed all.zarr at every ladder node, and for the sampled nodes its values against the k-way fold of the node's own {window}.zarr overviews, regime: stage-merge, merges_from_raw one more than the deepest of those overviews', source_windows equal to the windows the node holds, and source_children / generation equal to those overviews' blocks summed. A fold that consumed fewer windows than the node now holds is reported STALE. Where the store declares no all-time fold the check is reported SKIP — not applicable, never left out; the same nodes are still probed, and a committed all.zarr found anyway (debris of an earlier declaration — the pyramid block is not frozen) is named under "NOT validated".

What it reads on a windowed store. The leaf roster from the run records (one LIST of the root and its stats_*.parquet objects; --roster list walks the store instead, --roster moc takes the shards from the root coverage.moc and one LIST per shard for its windows); for the roster's cross-check one LIST of the root, one per order-0 node (at most 12) and, under the run records, one GET of coverage.moc; one zarr.json GET per declared (node, window) overview, per (leaf, window) column and per ladder node for its all.zarr (declared or not) — the unwindowed bound times the window count; then array reads for the sampled windows × nodes × cells, and, per sampled all-time node, one LIST of the node's prefix and the sampled cells of each of its windows' overviews. Nothing is written.

Not checked, and said so. The staged sweep's stage columns above the shard (orchestration, not contract — specification §4.6) on either kind of store; a windowed zagg-pyramid/1 store (a legacy orders schedule, or a raster store), whose declaration fails naming that; and whatever the report lists under "check(s) declined — NOT validated".

The commit stamp

S3 has no empty directories and LIST is strongly consistent, so absence is trustworthy: a delimiter-LIST with no digit children means nothing finer exists. Presence is not — a worker that dies mid-shard has already created the .zarr/ prefix. So the shard's final write is a root group.attrs.update(...) recording completion (D4):

"morton_hive_commit": {
  "spec": "morton-hive/1",
  "complete": true,
  "cells_with_data": 412,
  "granule_count": 17,
  "written_at": "2026-07-10T12:03:41+00:00",
  "content_hashes": {"arrays": {"13/count": "…", "13/morton": "…"}, "combined": "…"}
}

content_hashes (issue #580) is the specification §5 O11 record — per-array sha256 over decoded values plus the combined digest — computed from the arrays the worker just wrote, before the stamp lands, so the stamp certifies the digest of what it seals. The D20 stats sidecar carries the same record; a leaf stamped before the key existed is unverifiable from the stamp alone, never tampered. A windowed leaf's record has no {cell_order}/morton key: that leaf stores no coordinate array (see Time windows).

The sidecar (and the run parquet row flattened from it) carries two clocks (issue #589): duration_s is the worker's read + index + aggregate wall — it keeps that meaning because every existing pin and readout reads it — and duration_total_s is the whole unit's, handler entry to the record, which is what gb_seconds / est_cost_usd price from. phase_timings.{write,hash,column,icechunk} (the run parquet flattens them to phase_*) are the write side between them. On the sharded leaf path (the fleet's) they are all stamped after duration_s stops; the unsharded streaming path writes from inside process_shard, so there the two clocks overlap.

A leaf whose root metadata lacks the stamp is debris: incomplete, ignorable, safe to overwrite on retry (the writer re-emits the leaf template with overwrite=True, so retries are idempotent). This is not consolidated metadata — one small PUT rewriting the root zarr.json, which the leaf template creates anyway. A shard that errors, or streams no chunks (no data), leaves no stamp; a fully empty shard leaves no .zarr/ prefix at all (the leaf is created lazily on the first chunk write).

The stamp also carries the shard's coverage envelope — see Coverage below. The sidecar it points to is written before the stamp, so coverage shares the debris semantics: no stamp, no visible coverage. A windowed leaf's stamp (Time windows) declares spec: "morton-hive/2" and adds window (the label) plus time_range — the actual [t_min, t_max] written, as ISO-8601 UTC strings.

Versioned leaves (specification §1.5, issue #582). The stable {id}.zarr/zarr.json is a pointer stamp: it carries the stamp above plus "current": "run-{run_id}-{attempt}", and the arrays live in that version subgroup — {id}.zarr/run-{run_id}-{attempt}/{cell_order}/…, a complete leaf with its own stamp, never rewritten once stamped. That stamp keeps spec morton-hive/1 or /2 exactly as above (windowed ⇒ /2): no spec value marks versioning — current alone does. A replacement writes a new version and swaps the pointer (one PUT), so an earlier run tag in the Icechunk repo keeps reading the base-leaf version it indexed (columns and overviews are still rewritten at their keys until issue #584, so a tag reads those only as the latest run left them); superseded versions are reclaimed by the operator-run collector (tools/icechunk_gc_targets.py, dry-run default: uv run python tools/icechunk_gc_targets.py <store_root> [--execute] [--anon] lists each versioned leaf's versions that are neither current nor referenced by any retained snapshot — every branch's and every tag's ancestry — and not in flight (a stamped version newer than the newest run- tag's finalize, or an unstamped attempt of a run without a tag, is kept), prints the reclaimable bytes, and deletes them only under --execute; a legacy leaf is never a target and a store without a repo is refused). Write order: (1) version arrays → (2) version stamp → (3) refs against the version's objects (a commit under commit: "leaf", the ladder sidecar under commit: "ladder") → (4) pointer swap. A path reader sees the previous state until (4); an Icechunk reader sees the new version at its commit — (3) per leaf, the staged sweep's node commit (possibly after (4)) under the ladder. Version objects are never touched: the skip-path lifecycle touch of a versioned unit refreshes the root zarr.json and the unit's siblings only (The lifecycle touch), so an expiration rule MUST NOT cover version subgroups. attempt is a per-invocation nonce (8 hex characters of a fresh uuid4), so two writers of one unit — a duplicate-invoke retry, a redundant fleet worker — never share a prefix: a retry always writes a new version, and an attempt that died before its pointer swap is left for the collector. The run tag run-{run_id} still groups all of a run's versions by prefix. A stamp without current is a legacy leaf (every store written before this revision): readers open the root and follow current when present, else read the root itself (zagg.hive.resolve_leaf) — no migration, and a store mixes both kinds after its first post-revision write. A root stamp naming current over a root that also holds {cell_order}/… arrays is a converted legacy leaf: the root arrays are the last legacy write, read only by readers that ignore current. A pointer naming a missing or unstamped version is a corrupted leaf, read as debris; a writer about to clear-and-template a root whose stamp names current MUST refuse (it is a legacy or stale writer against a versioned leaf). The root's mirrored content_hashes keys are relative to the version root, identical in form to a legacy leaf's. Operator preconditions (normative, spec §1.5). Versioned leaves are the hive default (output.leaf_versions: false opts a run out). A store written with them MUST carry no bucket expiration rule over its hive tree (the collector owns version lifetime — see The lifecycle touch), and its readers MUST follow current (moczarr takes that change before the next fleet run).

Reader caveat — t_max floors, so the recorded range can end up to 1 s early. Both ends render through windows.iso_utc's whole-second granularity (isoformat(timespec="seconds")), so each truncates to the second containing it. On t_min that is harmless: the recorded start is at or before the true first observation. On t_max it is not — a last observation at 12:00:00.7 is recorded as 12:00:00, so the closed [t_min, t_max] is a slight under-estimate of the extent written, not an envelope around it. A consumer that prunes leaves on the stamp must treat t_max as inclusive with 1 s of slack (test the query against t_max + 1 s); pruning on the recorded value alone can skip a leaf that genuinely holds data in the queried interval. Sub-second inputs reach the rendering on both pipelines — a float delta_time column on the point path (windows.iso_time_range), a millisecond-bearing STAC datetime on the raster path (processing/raster._us_iso). The narrowing is recorded rather than fixed (ruled on PR #398): no in-tree consumer prunes on time_range — every use is a union into the regenerable root coverage.moc summary or the overview rollup, or a skip-if-current comparison between two identically-rendered values (sweep.MocFamily.finish, which rewrites the root object only when the union widens what is already recorded) — and rendering the range outward instead would change the bytes of a stamp field external readers decode.

Coverage

Where the data is, declared hierarchically (issue #200, design §4 as amended by PR #206; O8/O9 resolved on the issue thread). Three tiers per shard plus two store-root objects:

tier	what	where	cost to read
0 — morton box	canonical ≤ 4-member cover of the occupied cells (DCA children, each tightened)	coverage payload on the commit stamp	free — rides the stamp GET readers already make
1 — exact bitmap	zstd-compressed bit field over the shard subtree at cell_order	{full_id}.zarr/coverage.moc sidecar	one opt-in GET
2 — exact truth	the leaf's morton coordinate (a stored array; derived from the leaf id on a windowed leaf) and its data arrays	the leaf's data plane	array read; the tiers above are indexes, never truth (D9)
root	shard-order ranges MOC over all completed shards	{store_root}/coverage.moc	one GET — the discovery bootstrap
root sibling	§10.5 word-set cover: a per-shard toc word SET (temporal stores only)	{store_root}/coverage.toc	one opt-in GET, temporal consumers only, on demand
leaf record	§10.6 temporal record: the leaf's envelope word + counted cover (temporal stores only; written by the leaf's worker, never by a sweep)	{full_id}.zarr/temporal.toc sidecar (a versioned leaf's: in its current version, beside the bitmap)	one small GET — the sweep's per-leaf read; readers never need it

Leaf envelope (on the stamp, zagg.hive.read_coverage; strict spec: morton-moc/1 gate — unknown specs read as absent):

"coverage": {
  "spec": "morton-moc/1",
  "box": ["-42113221", "-42113224", null, null],
  "cell_order": 12,
  "source": "worker",
  "encoding": "bitmap",
  "sidecar": "coverage.moc",
  "nbytes": 213,
  "raw_nbytes": 512
}

box is always exactly 4 slots, nulls trailing; members are D1 decimal strings. encoding/sidecar/sizes appear only when the bitmap exists — a box-only envelope (phase-1-era leaf, or a depth-0 child_order == parent_order config) is read as "box only". No generated_at: the stamp's written_at is the one clock and one writer. Bit convention (frozen with the mortie-side spec): bit i = the i-th shard-subtree cell in ascending packed-word order (base-4 value of the D1 digit tail, digits 1..4 → 0..3), MSB-first per byte. A corrupt sidecar (bad zstd, wrong size) raises; a missing one degrades to None — a truncated bitmap must never read as a plausible partial cell set. The sidecar is the one foreign key inside the otherwise-vanilla leaf: zarr data reads are unaffected, but member enumeration (members()/tree()) emits a ZarrUserWarning and skips it.

Root envelope ({store_root}/coverage.moc, zagg.coverage.load_coverage):

{
  "spec": "morton-moc/1",
  "encoding": "ranges",
  "order": 6,
  "source": "dispatcher",
  "generated_at": "2026-07-10T22:59:35+00:00",
  "ranges": [["5112333", "5112333"], ["-4211321", "-4211324"]]
}

The example above is zagg.hive.build_root_coverage output for the shards -4211321..-4211324 plus 5112333 (all order 6) and round-trips through root_coverage_words; the test suite parses it straight out of this file so the reference example can never drift from the implementation.

A temporal-declaring store adds one more key here: temporal, the zagg-coverage-toc/1 section (per-shard toc envelope words plus an optional root counted cover) whose grammar is normative in specification.md §10 — one metadata GET then answers "which shards hold data DURING my window" before any leaf is opened. A store with no temporal channel carries no such key and its root object is byte-identical to a pre-#480 one; absence is never a refusal.

A range is an inclusive run of same-order cells within one base cell, consecutive in digit-tail rank; endpoints are decimal strings (packed u64 words exceed 2^53 and raw JSON numbers get mangled by float-based parsers). source is "dispatcher" (end-of-run write) or "refresh" (the explicit walk rebuild); the sweep will add its own.

The same store may carry one more root sibling, {store_root}/coverage.toc (issue #489): the §10.5 word-set cover — per shard, a small canonical toc word SET instead of the section's single envelope word, so a window that falls in a gap between a shard's observation campaigns prunes that shard too. It is GET-on-demand by temporal consumers only (at CA scale ~1.5 MB against the ~0.1 MB tier-1-carrying bootstrap object it sits beside — ~15×, and ~300× a spatial-only pre-#480 root — which is why it is not inline), discovered through the section's cover marker, and carries the same regenerable-accelerator staleness posture as everything else on this page. Grammar: specification.md §10.5.

Leaf temporal record ({full_id}.zarr/temporal.toc, issue #575): on a temporal store every leaf the worker writes also carries a small JSON record — its §10.2 envelope word, its §10.3 counted cover (observation counts per aligned time bucket) and the §10.5 cover derived from it — computed per chunk from the toc words the aggregation already encodes, and PUT in the bitmap's slot: after the arrays, before the stamp, fail-closed — a record that does not land fails the unit before the stamp (below). It exists so the families sweep composes the root section and the cover sibling from one small GET per leaf instead of reading every leaf's raw _times column back (a million-row ragged array per field per leaf at California scale, which no single invoke could finish). On a versioned leaf the record (like the bitmap) lives in the version subgroup — {full_id}.zarr/run-{run_id}-{attempt}/temporal.toc — and every walk resolves current before reading it. On a windowed store every window leaf ({full_id}_{window}.zarr) carries its own record, counted from that window's observations alone, whichever unit wrote it: the bulk per-shard invoke (issue #586) folds one accumulator per window — fed that window's binned observations on the pooled path and in both spill regimes — and writes each leaf's record before that leaf's stamp, so the record is the one unit: window writes, byte for byte. A record that cannot be built or written is its own window's failure: that leaf stays unstamped, the shard's other windows land with theirs, and the run's tail sweeps and covers the windows that landed (the "post-run tail reads leaves" rule above).

The record is the worker's, or there is none. It is the only source of observation counts: the leaf stores no per-observation clock, so nothing can rebuild them afterwards. The sweep and the refresh escape hatch (zagg.coverage.refresh_root_coverage) read the record first and never write one — not for a leaf that lacks it, not over an unparsable, inconsistent or foreign-revision one. A record that passes its checks is used whatever its fields list says (issue #600): the list is provenance — the counts come from one shared clock column, so they do not depend on which fields the store declares — and is never compared with the manifest's declaration. A leaf without a usable record is read from its raw columns one chunk at a time and contributes its coverage only: its envelope word and its occupied buckets, every count zero (§10.3). So that nobody mistakes the result for a total, the root counts block carries uncounted_shards — the number of shards with such a leaf — and its obs_total is exact at 0 and a lower bound otherwise. The sweep record shows the same thing per pass: temporal_routes: {records, raw} (how each contributing leaf was read; raw leaves gave no counts) and, on a pass that wrote the section, uncounted_shards beside temporal_shards — both this pass's tally, not the root's: a pass over part of the store keeps the standing counts block and its marker (§10.4), so whether the published totals are exact is read from the root block's own uncounted_shards.

The root marker counts and does not name. Which shards came in uncounted is in the sweep record (issue #598), beside temporal_routes and on every pass of a temporal store that visited a leaf — in-process, in the Lambda sweep mode's response, and in the durable sweep_stats_{ts}.json — whether or not the pass wrote the root object:

"pass_uncounted": {"count": 1, "shards": ["11214"], "truncated": false}

• shards — the decimal ids, in the root map's key order, of the shards this pass visited that hold a leaf read raw. A shard the pass dropped from the section (a window leaf that did not read) is not in it, exactly as it is not in the marker.
• count — how many there are, always exact.
• truncated — true when shards was cut at 1,024 ids (zagg.sweep.UNCOUNTED_LIST_CAP): a store written before the record existed has every shard uncounted, and the record stays small.

It is the pass's own tally, like temporal_routes: a partitioned pass writes no root object and names only the shards of its partition (its sweep_stats_{ts}_p{index}of{of}.json is one of of such lists), and a pass over part of the store names only what it read. It is not the standing list behind the root marker — no object holds that. Only the sweep names: zagg.coverage.refresh_root_coverage rebuilds the section and sets the marker but writes no run record, so after a refresh the names take a partition-less sweep over the store's leaves, which names every shard it reads raw, up to the cap. Each named shard has at least one leaf to rewrite (a forced rewrite of that unit, whose worker writes the record).

There is no backfill for a store written before the record existed (espg ruling of 2026-10-01 on issue #575): such a store is frozen and rebuilt in the current layout, where every leaf gets its record at commit. Swept as it stands, it publishes tier 1, the cover, and a counts block of zero-count buckets with every shard uncounted — and pays the raw column read per leaf on every pass, since nothing converges, which at California scale no single invoke can finish (_handle_sweep in deployment/aws/lambda_handler.py works that arithmetic).

A record that fails to write fails the unit (espg ruling of 2026-10-01 on PR #578). The skip-if-current gate compares the semantic hash and the granule-id set and verifies the leaf column, but never looks for the record, so a leaf stamped without one would be skipped as current on every re-run and stay uncounted for good. The worker therefore raises before the stamp, after the object-store client's own retries (up to 12, backed off 1–30 s inside a 180 s budget — zagg.store._S3_RETRY_CONFIG; there is no second loop on top). A record that cannot be built fails the unit the same way, before the stamp, with its own error and no store call:

• a legacy leaf is left an unstamped prefix (arrays and bitmap, no record, no stamp) — debris, cleared wholesale by the next attempt's template. As with any failed legacy write, the previous committed leaf is already gone: the template cleared it before the first chunk;
• a versioned leaf's attempt is left an unstamped version subgroup; the pointer is not swapped, so the previous current (if any) keeps serving, and the retry draws a fresh version;
• no stats sidecar, granule-id sibling, sub-map or leaf column is written for the attempt, so a later run's gate sees nothing of it and does not skip the unit on its account: a legacy leaf is unstamped and rewritten; a versioned leaf is judged on its previous current alone (rewritten unless that version already holds the planned identity), and a unit with no previous version has no sidecar and is processed;
• on the fleet the invoke returns the handler's 500 envelope ({"error": "Unhandled exception: leaf temporal record for … failed to write …", "shard_key", "request_id"}) and its .status object is failed. A failed status is terminal for that run — the poller never re-fires a worker-reported error, and the fleet's async retry count is 0 — so the shard is counted in the run's errors and re-done by the next run.

A leaf whose fold saw no clocked observation writes no record and stamps; that is not a failure. A raw leaf now means one written before the record existed, one whose record was lost or damaged after the stamp, a revision or field-set mismatch, or a record read that failed on that pass. Grammar: specification.md §10.6.

Reader flow (zagg.coverage): load_coverage → root_coverage_and against the AOI to pick candidate shards (one GET, no walk); per leaf, box_and on the stamp payload for the cheap reject, then bitmap_and for exact cell-level filtering (falls back to the box verdict with None when the leaf is box-only), then the morton coordinate as truth. The box is a conservative superset — false positives cost one wasted read, false negatives are impossible; the bitmap and the root MOC are exact for what they list.

Staleness (O7): readers trust silently on the hot path. The root object is written fail-open at end of run while leaves stamp continuously, so the most common gap is benign — a run still in progress. Beyond that, a crashed run, an out-of-band write, or the benign concurrent-run union race (GET-union-PUT is not atomic; last writer wins until the next re-union) leaves it missing shards, which degrades to "reader doesn't see the newest run", never a wrong answer. zagg.coverage.warn_if_stale implements the lazy detection lean: when a reader opens a commit-stamped leaf the root MOC doesn't list, it warns once per store and suggests zagg.coverage.refresh_root_coverage — the explicit delimiter-LIST walk that rebuilds the root MOC from the stamped leaves (debris excluded) and writes it with source: "refresh". No reader ever auto-walks (D10).

Deploy note (the sync-invoke analogue is the mode: "ping" preflight, which replaced the PR #205 setup echo — issue #252): the Lambda leg posts one fire-and-forget mode: "coverage" invoke, which requires the redeployed function. An older deployment 400s the event in its process handler — a logged error line in CloudWatch, but no writes, no result mirror, and no async redelivery — so the failure is fail-open by construction; the root object simply doesn't appear until the sweep or a refresh builds it.

Migration: the D19 hash epoch

Every semantic_hash written before this release is invalidated, by design — and so is every granules_sha256 taken over resolved hrefs (item 5). Nothing in any store changed on disk; what changed is the derivation of the digests that label it. If you are upgrading a store built by an earlier zagg, read this section before rerunning into it.

Issue #415 closed two ruled defects in the semantic core (PR #397 questions (7) and (8)), and carried two further ruled exclusions that had to ride the same epoch: the credential mechanism (issue #449) and the byte-movement knobs (espg-ruled 2026-08-17 on the epoch PR). Those four change what zagg.semantics.semantic_hash digests, so each moves every digest — which is why they were deliberately landed in one release rather than one at a time. A fifth ruling (item 5 below) canonicalizes granule identity, moving the sidecars' granules_sha256 rather than semantic_hash; it rides the same release for the same reason, since both halves of the identity PAIR are what the skip gate compares.

What changed

1. The granule fan-out width left the core. shard_workers / granule_workers are now in zagg.semantics.DATA_SOURCE_PACKAGING_KEYS. D19's ratified exclusion list already called worker size packaging, but the keys were never listed, so the digest moved with the pool width. That was not merely untidy: both dispatchers hand each cell a data_source clamped to min(K, n_granules), so a small shard's worker computed a different digest from the run's, and any rollup mixing clamped and unclamped shards collapsed semantic_hash to null.
2. The leaf-shaping output knobs entered the core — zagg.semantics.OUTPUT_LEAF_SHAPING_KEYS (aoi_mask, windowing) and GRID_LEAF_SHAPING_KEYS (sharded). Before, the whole output block was outside the core, so a config edit that changed what a leaf contains moved neither half of the skip gate's identity pair and a rerun read the stale leaf as current. output.grid.emit_cell_ids meets that criterion too and is still excluded (espg-ruled 2026-08-17): the D16 hatch is scheduled for removal (issue #304), and hashing it would leave every store built with it ON carrying a digest that no legal config can reproduce once the knob is gone. A leaf's array inventory is verified by reading the leaf — the same argument that keeps output.pyramid out.

3. The credential mechanism left the core. data_source.credentials_provider joined DATA_SOURCE_PACKAGING_KEYS (espg-ruled 2026-08-17): the provider name selects how source bytes are fetched, never what is computed from them, so the same granules read with lpdaac credentials, gesdisc credentials, or an anonymous open are one product. It is the same class as the read knobs and as anonymous, already excluded — the same class, not one knob under two names: anonymous is read only by the raster source-store kwargs, credentials_provider only by the point and temporal paths, so no single run consults both. A wrong credential fails the fetch loudly (a 403 at read time), so nothing depended on the digest to catch it. The operator consequence is the one that made it urgent: a credential migration over unchanged data — re-registering an existing store's source under a different provider, or adding the key to a config that ran without it — no longer refuses the store and rewrites it to produce the same bytes.

4. Three more byte-movement knobs left the core. read_workers, write_buffer and source_region joined DATA_SOURCE_PACKAGING_KEYS (espg-ruled 2026-08-17): read_workers is the third fan-out width beside the two spellings item 1 excluded, write_buffer bounds how many slabs are alive under the streamed raster sink, and source_region is the raster source store's AWS region kwarg — which sat in the same dict literal as the already-excluded anonymous, so hashing one and not the other split a single "how do we open the source" decision across both sides of the line. Each fails loudly in its own direction (a small pool is slower, a wrong region is a connection error, an over-large buffer is an OOM), so nothing depended on the digest to catch them. The operator consequence matches item 3: retuning machinery over unchanged data no longer rehashes. The live demonstration is dated, and it covers the fan-out widths only — two GEDI flux builds of the same shard on 2026-08-17 produced identical total_obs and cells_with_data in the exact single-block spill regime and still hashed apart on read_workers and the two *_workers spellings of item 1. It cannot cover the other two: write_buffer and source_region are read only on the raster path, and a GEDI flux build is the point path, so their exclusion rests on the arguments above rather than on this measurement.

5. Granule identity is now the driver-stripped bare granule id. This one moves the other digest — granules_sha256, the catalog identity half recorded in every D20 sidecar, and the id list in its granules.json sibling. espg-ruled 2026-08-17: "we want the granule to trigger the hash, not how that granule is fetched." A single granule is named three ways across the paths that record it — a resolved s3://bucket/key/FILE href, an https://host/path/FILE href (data_source.driver picks one), or the bare catalog id, which for every catalog zagg reads is the basename of both hrefs. Pre-epoch each spelling hashed differently, so a driver switch — packaging in the semantic core since forever — made every leaf's recorded catalog identity un-reproducible and sent the skip gate down the expansion arm for a rerun over exactly the same granules. Both halves canonicalize: the digest and the recorded id list, so the missing ids a contraction names are driver-independent too. Recorded lists written before this release keep working — the classifier canonicalizes the recorded side as well, so a pre-epoch leaf diffs cleanly instead of reading as a full contraction. Accepted cost of the ruling: two granules whose hrefs differ only in prefix collapse to one identity; every catalog zagg reads names granules globally uniquely, which is why the catalog's own id equals the basename. That cost lands in two places, not one — besides the identity collision it degrades the contraction guard inside a collapsed group, and the safe way is not the way it falls: dropping one member of a collided pair leaves the set diff unchanged, so the leaf reads id-multiset-drift and rewrites where the pre-epoch diff refused and named the dropped href. Any leaf whose recorded ids collapse is logged loudly for exactly that reason; making it refuse instead is a standing question on the contraction predicate (PR #420 review finding (2)).

Deliberately not changed: the orders (parent_order / child_order / chunk_inner) stay packaging — hashing them would make o8 and o9 runs different products and block mixed-order processing (D24) — and the whole pyramid block stays out, so retrofitting a pyramid declaration onto the config that built a store still hashes identically and still works. emit_cell_ids stays out for the reason recorded under item 2.

Why re-hashing is correct, not a defect

The pre-epoch digest answered "do these two configs produce the same leaves" wrongly in both directions: it separated configs that were identical in every effect (the clamp), and it equated configs that write different leaves (the output knobs). A digest that is wrong in both directions cannot be preserved for compatibility — preserving it would mean preserving a false answer to the only question it is asked. The stores themselves are untouched: no leaf byte moves, and the §5 content_hashes (which are over decoded values, not over identity) are unchanged. Only the label is restated, more accurately.

What an operator sees, and the three ways forward

semantic_hash is a frozen manifest key, so a rerun into a pre-epoch store refuses up front, before any leaf is written:

morton_hive.json at s3://bucket/store does not match this run
(existing {...} vs {...}); this store was templated for different
orders/identity — clear the store root (or pick a new one) before
writing with this configuration

--overwrite does not bypass it: when the digit tree already holds shard data, the overwrite path refuses too, because replacing only the manifest would leave the old leaves masquerading as legal data (D2). Pre-#299 stores that carry no hash at all are unaffected — the guard exempts a missing hash on either side, so they stay resumable exactly as before.

1. New product root (recommended). Write under a new output.product_name (or a new store root). Both products coexist under one root (D19), the old one stays readable, and nothing is rewritten or lost.
2. Rebuild. Clear the store root and rerun. Correct and simple; you pay the full aggregation again.
3. Restamp the manifest in place — expert path, read the whole entry. The data really was produced by this config, so writing the new digest into morton_hive.json is semantically correct. It is also the only path that can make things worse, in three ways:
4. It is not one field. semantic_hash is recorded in leaf attrs and in every D20 stats sidecar as well as the manifest, and dedup's identity checks read the sidecar's copy, not the manifest's. A manifest-only restamp therefore leaves a mixed-identity store — root says one thing, leaves say another — until every leaf has been rewritten. Nothing else in zagg expects that state to persist.
5. The rewrite is not a heal, it is the same re-aggregation as (2), paid unpredictably across future runs instead of once deliberately: each unit classifies semantic-mismatch and rewrites wholesale.
6. It manually defeats the guard that catches a wrong config. Nothing verifies that the config you hash is the one that built the store, and once a foreign digest is installed the frozen-key check will never fire for that store again. Before restamping, confirm the config is the store's own by recomputing its pre-epoch digest under the previous zagg and matching it against the recorded one.

There is no zagg tool for any of this; it is a deliberate operator action, and paths (1) and (2) have none of these failure modes.

Whichever path you take, the first post-epoch run over a store is a full rewrite of everything it touches — on a fleet-scale store that is a full re-aggregation, and it is the headline cost of the epoch, not a footnote. The skip gate cannot certify a leaf as current against an identity that was computed by a different rule, and pretending otherwise is exactly the false skip the epoch exists to prevent.

On the Lambda path the manifest write is asynchronous (issue #252 hybrid, above), but the refusal is not deferred with it: the read-only frozen-key precheck (zagg.hive.validate_manifest) runs on the mode: "ping" preflight before fan-out, so an epoch mismatch costs one preflight, not a few thousand worker invocations.

What the epoch buys

The break is the price of arming the fleet's leaf identity gate. Skip-if-current (below) is armed by default on the local backend and, since PR #581, on the fleet's point path; fleet arming was explicitly gated on this epoch (issue #415, sequencing), for the reason phase (7) records: pre-epoch the worker-side fallback hash was clamp-sensitive, so a small shard would have compared its leaf against a digest the run never wrote and rewritten forever without self-healing. After the epoch that comparison is sound, and the second post-epoch run over an unchanged store is the no-op the gate was built for.

Migration: the index-exclusion epoch (issue #499)

Every semantic_hash of a store whose config carried a data_source.index block stops reproducing — and both deployed stores do. Nothing on disk changed; the derivation of the digest did, exactly as in the D19 hash epoch above. Unlike that epoch, this one has a tool-driven migration and a store it applies to.

What changed (espg-ruled 2026-09-13; refs issue #499, issue #547, PR #565): the chunk-index block — data_source.index: backend (inline/hierarchical/sidecar), the sidecar store location, on_miss — is read machinery, the same D19 class as reader and read_plan, and now sits in zagg.semantics.DATA_SOURCE_PACKAGING_KEYS. A sidecar miss processes the file (on_miss: build populates a cache, it never filters an observation), and a different sidecar location yields byte-identical leaves. Hashing the block therefore split one product across every relocation of its index cache — and the relocation is exactly what #499 is doing: the sidecars move to a public bucket, and a digest that named the old location would have refused every append to the store they index. That copy landed 2026-09-17: the ATL03 sidecars are at s3://us-west-2.opendata.source.coop/englacial/zagg/demo/sidecar/ATL03/007/ (same keys, publicly readable, and inside the fleet execution role's englacial/zagg/demo/* grant, so the fleet reads and populates it; zagg's sidecar read still signs with ambient credentials, so a credential-less reader should drop the index block rather than rely on on_miss: fallback), and the packaged template reads from it. The table below keeps the store's build-time block, which is what its frozen digest was derived from.

Why re-hashing is correct, not a defect: the argument of the D19 epoch holds unchanged — the pre-epoch digest answered "do these two configs produce the same leaves" wrongly (a relocated index cache read as a different product), and a wrong answer cannot be preserved for compatibility. No leaf byte moves; only the label is restated.

Who it hits. Both stores that were live at the epoch were built with an index block, so their frozen manifest hashes are pre-epoch (the ATL03 v3 rebuild of 2026-10-08 was born post-epoch — see The live stores):

store	data_source.index	frozen semantic_hash today	migrates to
atl03_tdigest_o9	{backend: sidecar, store: s3://sliderule-public-cors/zagg-index/ATL03/007, on_miss: build}	b9b15fdde78f…	aacfe1e387d2…
gedi_flux_o9	{backend: hierarchical}	4f8287947a83…	337b2c3acac9…

(Full digests are pinned in tests/test_semantics.py::TestIndexExclusionEpoch::test_the_live_stores_migrate against the stores' vendored build configs.) A config with no index block hashes identically under both epochs, so no other store moves.

Two packaged templates do: atl03_tdigest_healpix_hive (1d818cf6… → 818bdd69…, its block {backend: inline}) and gedi01b_waveform_healpix_hive (c812c910… → b55a152c…, {backend: hierarchical}). The first now shares its digest with atl03_tdigest_healpix (818bdd69…): the default-valued index block was their only core-visible difference, so post-epoch the two templates name one D19 product identity — the ruling working as intended, not a defect. (Since issue #638 the hive template is windowed yearly and declares the delta_time column it bins on, both hashed, so as shipped the two name different products again; the collapse holds once those two keys are removed, which is what tests/test_semantics.py::test_the_two_atl03_templates_collapse pins.) No live store is built from either template's unmodified digest, so this is a record, not a migration.

The migration — one tool, one write. zagg.semantics.semantic_hash_legacy recomputes a config's pre-epoch digest (the current core with the index block re-inserted). declare_pyramid — the retrofit path behind tools/redeclare_dense_ladder.py and python -m zagg.sweep --declare-pyramid — consults it: a store whose frozen hash is the supplied config's pre-epoch digest is accepted (this config built the store) and the same manifest PUT rewrites semantic_hash to the current digest, even when the pyramid declaration itself is unchanged. That is the one place a frozen key ever moves. The same call re-renders the store's D19 core sidecar (aggregation.yaml) from that config — nothing else regenerates it, so a migrated store would otherwise go on showing a pre-epoch core (with its data_source.index block) beside a post-epoch hash. The tool's dry run prints the migration before --execute:

semantic guard: legacy MATCH (b9b15fdde78f) → will rewrite to aacfe1e387d2 (pre-epoch hash; ...)

The append path does not migrate. zagg.hive._frozen_matches compares current-epoch digests only, so an aggregation run into a not-yet-migrated store refuses up front, and the refusal names the tool:

morton_hive.json at s3://bucket/store does not match this run (...); the only
frozen key that differs is semantic_hash, and the store's is this config's
PRE-EPOCH digest (issue #499: data_source.index left the semantic core) —
migrate the manifest with tools/redeclare_dense_ladder.py (dry-run prints the
migration; --execute writes it), then rerun; the append path never migrates
on its own

Accepting legacy digests there would make every append a silent migration point and the manifest would never actually move; refusing by name keeps the operator's --execute the single event. Restamping by hand (path (3) of the D19 epoch note) is not needed and not recommended: the tool verifies that the config is the store's own before it writes.

Skip-gate consequence — deliberate, not fixed. zagg.dedup compares the hash strings stamped in run records and leaf sidecars against the current digest, and those stamps on the two stores in the table above (atl03_tdigest_o9, gedi_flux_o9) are pre-epoch. After the epoch they read as stale: a same-shard re-dispatch into either store rewrites the leaf instead of skipping it (the semantic-mismatch classification), exactly the D19 epoch's "first post-epoch run is a full rewrite of everything it touches". New-AOI appends (no prior leaf to compare) and the column backfill (which reads leaves, not stamps) are unaffected. Rewriting is the safe direction — the gate cannot certify a leaf as current against an identity computed by a different rule — and the manifest migration does not touch leaf stamps, so it neither causes nor cures this.

Re-runs: skip-if-current, the contraction guard, and the lifecycle touch

Re-dispatching a shard used to mean an unconditional wholesale rewrite (D4) — correct and deterministic, but wasted compute, and invisible to bucket lifecycle policies that purge "old" objects. Since issue #388 each (shard, window) unit runs a worker-side identity gate before any read or fold, on both leaf families (the aggregation seam zagg.hive.process_and_write_hive and the raster seam zagg.processing.raster.process_and_write_raster_hive).

Identity is a pair: the run's semantic_hash (D19 — what/how) × the unit's planned granule-id set (over what), compared against the leaf's recorded stats sidecar (fast path: one granules_sha256 compare). A matched pair alone is not sufficient — the unit's artifacts must be current too, verified by reading them, never trusted from the record: the leaf must carry its commit stamp, and the leaf column must agree with the run's declaration (the specification §4.6 config-decides gate: the declaration moves neither identity half, so the gate reads the artifact). Three verdicts:

verdict	when	what happens
current	both halves match, leaf stamped, column agrees with the declaration	fold no-ops; the unit writes nothing (no arrays, no stamp, no sidecar, no sub-map, no column — zero sweep dirtiness, with one exception: a unit whose touch moved its Icechunk ref checksums and rewrote its ladder ref sidecar enters the staged sweep as dirt-only, below); the lifecycle touch below runs; counted as cells_current
refused	the planned set drops recorded ids: recorded ∖ planned ≠ ∅ — deliberately not strict-subset, so a shardmap that grew while silently dropping old granules (an upstream purge behind a fresh catalog query) still trips it	the unit refuses, writes nothing, and counts as cells_refused — never as an error. The log names the first five missing ids and their total; the full per-unit diff is the refusal manifest at the store root (below). --allow-contraction (agg(allow_contraction=True); on Lambda an allow_contraction event field) turns it into a normal rewrite
rewrite	everything else — expansion (new cycles), a semantic change, no/unreadable sidecar, an unstamped leaf, column drift, or a pre-#388 sidecar (below)	today's wholesale D4 rewrite, column included

The gate is on by default for hive point runs on every backend; --overwrite disables it entirely (the operator's unconditional-rewrite hammer — it does not acknowledge a contraction, it bypasses the guard). On the fleet, both point-path dispatchers (agg(backend="lambda") and the client facade) arm it per cell event with skip_if_current: true plus the RUN config's semantic_hash, and the handler forwards both keys and allow_contraction to the shared seam; a current or refused unit then returns no stats record and writes no sidecar or sub-map, exactly as on the local backend (issue #401's clobber gate; PR #581). An event without the key — an older dispatcher — keeps the unconditional rewrite. Raster fleet units stay unarmed (the handler's raster body fix below is still owed). What fleet sidecars record splits by family:

• Vector — nothing to wait for. The handler passes the seam's own metadata dict straight to build_record, the seam stamps semantic_hash into that dict, and the record's validated fallback picks it up. So from this release's deploy onward — gate still off fleet-side, zero handler changes — fleet vector sidecars carry both identity halves, and a later local re-run over that store can skip.
• Raster — never, until the handler changes. The raster branch rebuilds its record body from a closed key list that omits semantic_hash, so raster fleet sidecars keep recording semantic_hash: null. That fails the fast path and classifies semantic-mismatch → rewrite on every run, with no self-heal until the leaf has been rewritten under a handler that carries the key. The consequence is local, not merely fleet-side: re-running a fleet-built raster store locally — where the gate is on by default — reports cells_current: 0 indefinitely until question (1)'s key-list fix lands.

The catalog half needs no enablement in either family: the granule-id sibling is written by the seam, not by the dispatcher that writes the sidecar, so every leaf a fleet worker commits from this release on records its input set — which is what the contraction guard, running in whatever process re-dispatches that unit later, diffs against.

Where the recorded id list lives. Not in the stats sidecar: in a sibling object beside it — granules.json (granules_{window}.json windowed), on the sidecar's own spec-keyed naming grammar — written by the leaf seam right after the commit stamp, carrying the ids and the granules_sha256 they hash to. Identity equality never reads it: that is the sidecar's hash compare, which is what keeps a fan-out dedup check (dedup.shard_status per shard, one GET) and the worker-side run-record assembly small. The list is fetched exactly once, only when the hashes disagree, and only to name which granules a contraction dropped — a pole shard's ~4,600 ids are ≈550 KB, and they would otherwise ride every one of those GETs. A sibling whose granules_sha256 does not match the sidecar's (a torn rewrite: one of the two writes lost) is rejected as unrecorded rather than trusted.

The contraction guard cannot protect leaves written before this release. That sibling arrives with issue #388. A leaf written before it has only the recorded hash — there is no recorded set to diff — so any granule-hash mismatch over such a leaf classifies unrecorded-ids and performs the silent wholesale rewrite the guard exists to prevent, contraction or not. (A config-only rerun — same inputs, changed semantic core — is not one of these: the hashes agree, so it reads semantic-mismatch without any sibling read, and never inflates cells_unrecorded.) The guard only protects a leaf after that leaf has been rewritten at least once under this release. These rewrites are counted apart (cells_unrecorded in the run summary; the CLI prints them as "rewritten with the guard inert") so an operator can see how much of a store is still unguarded; making them refuse instead is a standing design fork (PR #397 question (4)).

One identity caveat operators still hit: the identity pair does not cover output.grid.chunk_inner, which changes the leaf's object set through K while both halves hold, so changing it still needs --overwrite (espg-ruled 2026-08-17 as the bounded state — closing it costs part of D24; sharded itself is covered since the epoch, see the migration note item 2).

The driver-dependence caveat is gone, in two different strengths either side of the epoch. The recorded id space used to be the resolved href, so flipping data_source.driver between runs read as a full mixed contraction and refused per leaf. Since the epoch both sides are reduced to the canonical driver-stripped bare granule id (item 5 of the migration note), so:

• against a leaf written at or after the epoch, a driver switch over unchanged granules reads equal and skips — its recorded granules_sha256 is already in the canonical space, so the hash fast path matches and no sibling is read;
• against a pre-epoch leaf it does not skip, and cannot: the fast path compares the stored digest, which was taken over resolved hrefs, so it mismatches by construction and the leaf is rewritten once (expected at an epoch — the note's "every pre-epoch granules_sha256 invalidated" headline is this). What canonicalizing the recorded side buys is the direction of that rewrite: the sibling's hrefs reduce to the same bare ids as the plan, so the leaf reads id-multiset-drift/expansion and rewrites, instead of refusing as a spurious contraction. A driver switch over a pre-epoch store therefore no longer needs --allow-contraction to get past the guard.

The refusal manifest

A refused unit writes nothing — no leaf, no sidecar, no run-parquet row — so a run that refuses leaves the store byte-identical and the only in-band trace is a worker log line truncated to five missing ids. Since issue #388 a run that ends with cells_refused > 0 also writes one small JSON object at the store root:

refusals_{YYYYmmddTHHMMSSZ}_{run_id}.json

Timestamp-first like the stats_*.parquet run records (and outside their glob, so run-record discovery never mistakes one for the other). It carries the run context needed to act on it — run_id, write timestamp, the run's semantic_hash, the zagg version — and one entry per refused unit: its shard_key and window, the classification (contraction or mixed), and the complete missing_granules list, which is exactly what the guard read from that leaf's granule-id sibling in order to name the drop. Triage the diff from this object, not from the logs.

Two deliberate limits. A pure-skip run (nothing refused) writes nothing here and stays row-less — the counters and the leaf sidecars are the record of a no-op run, and an empty manifest per rerun would be litter. And the manifest is written by the local backend only: D8 keeps the Lambda dispatcher from writing to the store, and no once-per-run worker-side seam exists yet to carry it — the same gap, with the same resolution, as the store-root touch below (PR #397 question (10)). A fleet run's refusals are visible in its summary counters and worker logs, not in a durable object.

The lifecycle touch

A skipped unit still resets the purge clock (except on a published bucket — see below): every object in its footprint gets a fresh LastModified (lifecycle rules act per object) — the leaf .zarr tree (stamp, arrays, in-leaf coverage.moc), the stats sidecar, its granule-id sibling and the sub-map sibling, and the declared column tree plus its own sidecar.

On a versioned leaf (Versioned leaves, spec §1.5) the footprint is exactly: the object {id}.zarr/zarr.json (the pointer stamp — never {id}.zarr/ as a tree), the stats sidecar, the granule-id sibling, the sub-map, the Icechunk ref sidecar, and the declared column tree plus its sidecar. No version subgroup — current or superseded — is touched: its objects are write-once and carry the checksums the repo's refs pin, so a self-copy (a re-minted multipart ETag) or an os.utime would break every snapshot and run tag that indexes them. Their lifetime belongs to the collector (tools/icechunk_gc_targets.py), not to bucket expiration, so an expiration rule over the leaf tree MUST NOT cover version subgroups — a rule that does ages out the current version under a live pointer on the rule's clock, whatever the collector decides, and bounds every tag's readability by the rule's age. S3 lifecycle filters select by prefix, tag or size and cannot exclude a nested name, so a store with versioned leaves MUST carry no expiration rule over its hive tree (spec §1.5's operator precondition; the icechunk/ exclusion below is the same posture for the repo). Local stores use os.utime; S3 uses a server-side self-copy (CopyObject onto itself, MetadataDirective: REPLACE) that preserves content, the ETag of non-multipart objects, and the object's storage class. A local run's wrap-up also touches the store-root trio no unit owns (morton_hive.json, aggregation.yaml, root coverage.moc) — an all-skip run re-PUTs none of them, and the manifest is REQUIRED reader-facing schema that would otherwise expire first, bricking a store whose data objects are all fresh.

The touch is best-effort and fail-open: a failed touch logs, counts, and degrades to today's behavior — it never fails the unit and never un-skips it. Watch the counters: objects_touched / touch_failures ride the run summary, and the CLI warns explicitly when touches failed, because those objects did not get their purge clock reset (the run still exits 0).

A published store is not touched at all. A path on a bucket in zagg.store._PUBLISHED_BUCKETS — Source Cooperative's us-west-2.opendata.source.coop today — is not applicable rather than touched or failed. The touch exists to defeat an expiration rule and an archival published bucket has none; worse, that bucket is versioned, so the self-copy does not refresh a timestamp in place — it writes a new full-size version and demotes the old one to noncurrent, where it keeps consuming storage. A single full-skip run over the CA store would add roughly 332 GB of noncurrent versions, invisible to ListObjectsV2, on a bucket AWS pays for as an Open Data sponsor. So a published store reports objects_touched: 0 with no failures, plus a touch_skipped_paths count naming how many footprint paths (not objects) were skipped — that key is present only when something was skipped, and is what tells a published run apart from a touch that never ran. The guard keys on the published set specifically, not on "external target": an injected-credential collaborator bucket has unknown lifecycle configuration and must still be touched, or the touch's whole purpose fails there.

S3 caveats, documented rather than solved: an object already transitioned to GLACIER/DEEP_ARCHIVE cannot be self-copied (counts as failed); ACLs are not preserved (moot under BucketOwnerEnforced, the modern default, but a public-read-by-ACL bucket would see touched objects go private); under SSE-KMS the copy re-encrypts with the bucket-default key and needs kms:Decrypt + kms:GenerateDataKey. Versioning is no longer on this list: a versioned bucket mints a new object version per touch, and that is the reason for the published skip above rather than a cost accepted.

Known gaps — plan lifecycle rules around them, they are not promised away: the design §7 sweep's ancestor-node artifacts belong to no unit's footprint and a skip produces zero sweep dirtiness by construction. That is the whole ancestor layer, not the overviews alone — all four rollup families age identically (stats.rollup.json, moc.rollup.json, submap.rollup.json, overview.rollup.json at every digit node, one per sweep.DEFAULT_FAMILIES entry), plus each pass's root sweep_stats_{ts}.json run record. And re-running the sweep does not refresh them: a second sweep over an unchanged tree recomputes but PUTs nothing, so the obvious "just sweep periodically to keep them alive" workaround silently no-ops. The store-root trio, separately, is touched by the local backend only — as is the refusal manifest above, for the same reason (D8 keeps the Lambda dispatcher from writing to the store, and the handler has no once-per-run root-write mode yet; both wait on the same resolution — PR #397 question (10)). The Icechunk companion repo (icechunk/, below) is outside the touch contract as well: touch_current_unit assembles a unit footprint of leaf tree + stats sidecar + granules.json + sub-map + declared column + the Icechunk ref sidecar (icechunk_refs.json, the ladder's input for that leaf — a skipped leaf must keep it as fresh as the leaf it describes, or the next staged sweep gathers a hole), touch_store_root covers the root trio, and neither reaches the repo. On a legacy leaf the touch does move the checksum every ref into the unit carries (spec §11.3: the file:// mtime; the ETag of a multipart-uploaded S3 object, which a self-copy re-mints), so a touched unit re-plans its refs from fresh HEADs: under commit: "leaf" it commits them at once; under the ladder it rewrites its ref sidecar and is marked icechunk_dirty, and the same run's staged sweep (output.sweep: "stages") takes it as a dirt-only leaf — in process on the local backend, and on the fleet as the stage event's dirt_only refs, which the dispatcher assembles from the worker bodies (D8) and the handler forwards to the stage worker: its nodes re-gather and commit their refs, but no overview or column is re-folded, since no data changed (each stage row counts them as icechunk_regathered; PR #581 question (11), ruled (a)). This narrows the #388 contract: a current unit writes no stats record, no sidecar and no sub-map, but it may enter the sweep work set as dirt-only when its touch moved ref checksums and the repo is on. A versioned leaf never does: its touch reaches no version object, so no ref checksum moves and the dirt-only re-gather applies to legacy leaves only. The ref sidecar's touch is unconditional: it is issued even when output.icechunk is off or the run used commit: "leaf", where the object does not exist — harmless (an absent sibling is neither touched nor failed), at the cost of one extra request per unit on an all-skip rerun. The repo is not a root-trio-style self-copy candidate either: it is many objects (snapshots, manifests, chunk-ref manifests, the branch ref), so touching it is O(objects in the repo) rather than three known keys. Nor does it degrade gracefully the way a missing sidecar does — lose one snapshot or manifest and the branch tip is unreadable, i.e. the whole order's index goes, not one leaf's refs. The operator lever is therefore a prefix-scoped exclusion on icechunk/ in the expiration rule (one prefix, one rule, no per-shard fan-out), not a writer-side touch; the index is regenerable by a full rerun, but nothing in stage 1 regenerates it on its own (issue #580).

There is no lifecycle rule that separates leaves from the ancestor artifacts above them. An S3 lifecycle filter keys on prefix, object tags, and size only; zagg's writers set no object tags; and per the specification §4.1–§4.2 an overview lives at an ancestor digit node inside the same prefix tree as the leaves it summarizes, where "nothing about the name distinguishes an overview from a leaf" (classification is by root-group attrs, which no lifecycle engine reads). So "scope expiry to the leaf data planes" is not writable as a rule: the only prefix that selects leaves while excluding ancestors is a per-shard-node prefix, against a 1,000-rule bucket cap. The honest options today are:

1. No expiry rule on a pyramid-declaring hive store. The safe default while the gaps above are open.
2. Accept expiry over the digit tree, understanding that it takes the ancestor artifacts with it. Those are regenerable caches, never load-bearing (specification §4.1), so the cost is a full rebuild by a sweep — not data loss. Exclude the store-root trio by name (three known objects at a known prefix); morton_hive.json is REQUIRED reader-facing schema and expiring it bricks the store.
3. The levers that would make a leaf-scoped rule writable do not exist yet: object tags stamped at write time (nothing in zagg writes any), or a scheduled refresher that re-PUTs the ancestor layer — the staged sweep of issue #384 is the obvious candidate, but nothing about it promises a refresh today. Both are open; see PR #397 question (10) for the related fleet-side root touch.

The Icechunk companion repo

Every hive leaf commit also records the leaf's inner chunks as Icechunk virtual chunk references in one companion repository per store, with a group per level, at the store root — specification §11 is the contract (issue #580, stage 1: refs-only, additive; the leaves stay normative):

{store_root}/
  morton_hive.json
  coverage.moc
  icechunk/                       <- ONE Icechunk repo per store (reserved name)
     /19                           <- a group per LEVEL, named by cell order: the base leaves …
     /13                           <- … the leaf columns' declared member (one object per leaf) …
     /12, /11, … /4                <- … and one per declared overview order (one object per node)
     /window_start, /window_end    <- the row coordinate every level shares
  {sign+base}/...                 <- the digit tree, unchanged

The repo is one zarr hierarchy: a group per level, keyed by cell order — exactly the manifest's multiscales datasets plus the base (the columns' other members are sweep intermediates and are not indexed) — each carrying the artifact's resolution-group attrs — dggs included — with every array re-rooted on the whole sphere at that cell order — count, morton, every field, the ragged vlen arrays and their siblings — chunked at the object's inner chunk (the leaf's inner chunk for the base and the §4.6 inner chunk for a fine column level; the whole object, one chunk per node, for an overview level or a column level below 4^5 cells) and coded with the inner chain (no sharding_indexed), so icechunk + zarr open the hive as ordinary arrays without moczarr.

Every level array is (n_rows, n_cells), chunked one row deep (zagg-icechunk/2, issue #584, spec §11.2). A row is one window of the store's schedule, or the reserved row all; an unwindowed store has the single row all, so its arrays read as array[0, …]. The repo root's zagg_icechunk.rows lists the row labels in allocation order — row w is rows[w], appended to in order of first appearance and never reordered, so a reader looks a row up by label rather than assuming chronological order — and two root arrays, window_start / window_end (int64, the manifest's temporal epoch and units; the fill value on the unbounded all row), carry each row's half-open time range. The level groups' dggs block (copied from the artifact, so spatial_dimension: "cells", coordinate: "morton") binds to each row slice array[w, …]: pick the row, then read it as the block says — morton holds the same words in every row that has the node's object. A repo written before the row dimension (zagg-icechunk/1) is refused by the writer with the remedy in the error: clear {store_root}/icechunk/ and let the next run's init re-create it. In a browser, icechunk-js reads the dense arrays today — morton, count, the per-field summaries; the zagg-ragged/1 vlen-bytes arrays await zarrita codec support (no vlen-bytes codec, and a closed dtype union — issue #580, zarr-extensions#71). The Python pair decodes both. The repo's root attrs mirror the manifest's zagg-multiscales/1 block as multiscales, so one open discovers every level (the icechunk-multiscales convention gridlook's level resolver reads). A leaf at shard rank r owns chunks (w, [r·C, (r+1)·C)) of its row w (spec §11.3); absent inner chunks emit no reference and read as fill; every reference carries the object's checksum in the form its container validates — the ETag on S3, the object's last_modified ceiled to the next whole second on a local store — so a leaf replaced in place fails loudly instead of decoding stale offsets (locally, only a replacement landing inside the same second slips through).

Four writes, all worker-side (the dispatcher never writes, D8), all fail-open — the repo is a regenerable index, never load-bearing:

• mode: "icechunk_init", one synchronous invoke after the ping and before the fan-out (the local backend calls zagg.icechunk_refs.init_repo in-process after the manifest lands): creates-or-opens the repo with every level group — the base, the column's declared member and one per declared overview level, keyed by cell order — defines their array nodes, one manifest split per group, the virtual chunk container and the multiscales mirror, allocates the run's rows (the event's rows, ["all"] for an unwindowed run: each new label grows every array by one row, in order of first appearance) and commits init {run_id} (every run — empty when neither the block nor the rows change, labelled split ratchet {from}->{to} {run_id} when it re-cuts, so the ancestry brackets each run). Idempotent — a rerun reopens; a repo built for another geometry, container or spec revision is refused, while split_order follows the ratchet below and commit / commit_order are per-run, never compared. The record (path, snapshot, created, options, levels, ladder, split_ratchet, rows) rides the run summary under icechunk; a failed init records {"error": …} there and the run proceeds refs-less. The run parquet broadcasts it as icechunk_init_repo, icechunk_init_snapshot, icechunk_init_error and icechunk_split_ratchet, always written so the column set does not vary run to run.
• The leaf's ref sidecar: after the leaf's stamp — last in the unit, behind the granule-id sibling and the issue #383 column fold — the worker sizes the objects it wrote (one HEAD + one ranged GET of the shard-index suffix per sharded leaf array and per fine column array; one HEAD per single-chunk column array below 4^5 cells, whose ref is the whole object) and writes the plan as a JSON sibling beside the leaf's stats sidecar (icechunk_refs.json, ≈40 KB at production geometry). No Icechunk session on the leaf path. The outcome rides the leaf's stats sidecar as icechunk — {sidecar, bytes, refs, arrays, checksum}, {skipped: "empty"}, or {error} — and the run parquet as icechunk_* columns. A windowed leaf is indexed like any other, at its window's row (spec §11.2). Its cost is its own phase_timings["icechunk"], not write.
• The ladder commits (zagg.icechunk_ladder, hooked into every node of the staged sweep): a stage node gathers its subtree's carriers — leaf sidecars at the finest tuple (one per (leaf, window) on a windowed store, each naming its own row), its children's node ref columns above — adds the refs of the overview objects at its tuple's orders, and either writes its own node column or commits. On a windowed store the hook runs once per node, after that node's window units and its close, so each row's refs are gathered and committed together. The tuple whose order range contains commit_order commits everything gathered in one commit per node covering every order in its subtree (node {decimal}, refs into /19/…, /13/…, /12/…, … of the one repo); coarser tuples commit only their own overviews; finer tuples write columns only. So a base manifest — one per split_order cell — is written by exactly one commit; the coarse levels hold the same number of chunks per manifest (one order-2 cell at /13, one base cell at /11 and coarser), so those few manifests are shared across nodes and rebased, which keeps the manifest count at the base's (spec §11.5). Each stage row records icechunk_commits, icechunk_rebases, icechunk_replays (commits replayed in a fresh session because another run's init grew the row axis under the node's staged refs, spec §11.4), icechunk_commit_s, icechunk_refs, icechunk_missing, icechunk_failed, icechunk_skipped_levels, icechunk_clean, icechunk_regathered (dirt-only nodes, re-gathered without a fold) and icechunk_s (the hook's own wall time, the gather included), plus an icechunk_nodes list naming each node that did work and the snapshot it committed; the first fleet run's contention question reads icechunk_rebases off the stage records.
• mode: "icechunk_finalize" (issue #582, spec §11.4), one synchronous invoke AFTER every commit of the run landed — after the staged sweep returned under the ladder (on both Lambda dispatchers, the CLI and the client facade: a sweep that did not complete leaves the run untagged, icechunk_finalize: {skipped}), after the fan-out under commit: "leaf" (the local backend calls zagg.icechunk_finalize.finalize_repo in-process at the same point): applies the retain_runs retention below, makes one content-free finalize {run_id} commit whose metadata names the run (run_id, semantic_hash, zagg_version, the ladder knobs) and its retention counts, and tags it run-{run_id}. Idempotent (an existing tag is returned, nothing written); a §11.5 split ratchet this run applied is reported as rewrite_pending for the operator rewrite_manifests pass, never run here. The record rides the run summary under icechunk_finalize; the tag is the durable outcome (repo.lookup_tag("run-…"), ancestry(tag=…)). Run.attach fires it too, off the config's knob and only for a pinned commit: "leaf" run with no sweep: "stages" (a sweep: "stages" run's is its dispatcher's, after the staged sweep), with newest_only: true (written only while the run is the newest on the repo, else {skipped}) and icechunk_init: null, so rewrite_pending is always null there. The operator's finalize (below) fires it with operator_checks: true and no config: the worker reads the run's dispatch manifest for the config and runs the operator's checks before a newest_only finalize, answering a failed check with {ok: false, refused: <reason>}.
• Operations (spec §11.4 Operations, issue #582) — the operator's way to change what the repo is authoritative for, one validated commit each, no leaf touched, the commit metadata naming the operation:

python -m zagg.icechunk_ops <store_root> set-attrs /19 '{"dggs": {...}}'   # root "/", a level "/{cells}", an array "/{cells}/{array}"; null deletes a key
python -m zagg.icechunk_ops <store_root> declare-pyramid config.yaml     # levels + multiscales follow the manifest's declaration
python -m zagg.icechunk_ops <store_root> finalize <run_id> [--function-name <fn>]   # tag a completed-but-untagged ladder run (the repo's newest run only); an s3:// store is finalized by the worker

Before the commit the array model of every array must be unchanged — but for row growth: new labels appended to rows, every array holding that many rows — and the block's compatibility keys must hold, else the session is discarded. set-attrs refuses the root's zagg_icechunk and multiscales keys; declare-pyramid adds a newly declared level's group (and its manifest split), delists a no-longer-declared one without deleting its group, and refuses a geometry change (that is a new spec revision). The manifest retrofit runs declare-pyramid itself (above). finalize (issue #588) is the repair for a ladder run whose dispatcher died after the staged sweep completed but before its finalize (icechunk_finalize: {skipped} on the CLI summary / handle.icechunk_finalize on the facade, or no record at all): it reads the run's dispatch manifest for its config (retain_runs, the semantic hash — no --retain-runs; a large hive run's manifest is slim — its block did not fit the setup event, so it rode without the shard list, shards: null plus shards_omitted — and still carries the config, so the run is finalizable; only a run whose manifest write was lost, or whose block did not fit even slim, is refused and left to the next run's tag — the run summary's dispatch_manifest / handle.dispatch_manifest says which went out: full, slim or dropped), refuses unless the newest sweep_stats_*_stages.json that names the run (pipeline_run_id == <run_id>, issue #593) since the run's init commit (the repo's clock) shows a completed sweep, and tags newest_only — an older untagged run stays covered by the next run's tag. A record naming another run, or none, vouches for nothing: a sibling run's sweep or an unnamed --stages pass landing after this run opened cannot tag it, and a later unnamed pass does not block the run's own record either. No --force: an incomplete sweep is completed with python -m zagg.sweep <store> --stages --scope all --pipeline-run-id <run_id> first. Where it runs. On an s3:// store all of that is the worker's: the command fires one synchronous mode: "icechunk_finalize" invoke — {mode, store_path, run_id, newest_only: true, operator_checks: true}, no config — and prints the worker's report, so the manifest and record reads, the commit, the tag and the retention run under the execution role in the store's region. The operator's host reads nothing from the store and writes nothing: it needs Lambda invoke rights only (on a Source Cooperative store the worker role is the only writer), and nothing leaves the region but the report. The function is --function-name, else ZAGG_LAMBDA_FUNCTION_NAME, else process-shard, an empty value counting as unset — the dispatchers' own default for a config with no worker: block, though they take an empty value verbatim (the run config's worker: suffix is never applied, since the only copy of that config the command could consult is the run's dispatch manifest, and it does not read the store). The report names the function invoked as function_name; the command never finalizes from the host. A check that does not hold comes back as a refusal with its reason; a deployed worker older than this operation fails on the missing config before any write — deploy a current worker. A local store root finalizes in-process. The step-by-step recovery around this command — finding the run, Run.attach, the hand sweep, then finalize — is the operator runbook, Recovering a run whose launcher died.

Why the ladder, and the scale settings. Per-leaf commits do not scale: at the full-globe worst case (3,145,728 order-9 leaves) they are 3.1M commits, and — the real limit — an Icechunk snapshot lists every manifest, so one manifest per order-6 cell per array is ≈442k manifests, a 44 MB snapshot read on every open, rebase and commit (≈2.2 TB of snapshot writes per run). Balancing snapshot bytes (≈100 B per manifest entry) against per-manifest bytes (≈2.5 KB on disk per leaf-array) gives leaves-per-manifest ≈ 0.6·√N — ≈30 at California scale, ≈1,000 at the globe. Both knobs live under output.icechunk:

output:
  icechunk:                 # true / false / absent (default on for hive) also accepted
    commit: ladder          # default: ladder when the run walks it, else "leaf" (the per-leaf commit; below)
    commit_order: 6         # default: the finest staged-sweep dispatch node (shard_order − tuple_width)
    split_order: 6          # default: commit_order; must be >= commit_order and <= shard_order
    retain_runs: 0          # default: keep every run tag; K > 0 keeps the K newest (finalize expires + collects the rest)

setting	manifest = one cell at	chunks / manifest (base level group)	commits	snapshot
default (6 / 6)	order 6, 64 leaves	4^7 = 16,384	one per order-6 node	≈390 base + a few hundred coarse-level manifests at California scale
global (split 4 / commit 3)	order 4, 1,024 leaves	4^9	768 (the order-3 nodes)	27k manifests, ≈2.7 MB

split_order is a one-way ratchet toward coarser (spec §11.5): a run whose config is finer than the store's recorded value adopts the store's with a warning; a coarser value re-cuts new manifests from this run on and flags the run parquet's icechunk_split_ratchet for a later rewrite_manifests pass over the old ones. tuple_width is unchanged (3). An unset commit resolves to ladder only when the run walks it — the dispatcher chains the staged sweep (output.sweep: "stages", on every dispatcher: the CLI, the local backend and the client facade, issue #588) and the store declares a /2 ladder with at least one composable field (zagg.icechunk_refs.ladder_walks) — and to leaf otherwise, so no run writes sidecars nothing gathers. The Lambda dispatchers ship the resolved mode in the worker config. The default commit_order is a function of the shard order and the width (zagg.icechunk_refs.finest_dispatch_order).

retain_runs is the run-tag retention (spec §11.4): 0 keeps every run (the default — history only grows, every commit kept); K > 0 has each finalize delete the run tags beyond the K − 1 newest, expire the snapshots older than the oldest retained run's finalize and garbage-collect the repo objects nothing retained references (the cutoff is always a run tag's commit time, never "now", so no in-flight object is collected — but a writer session opened before the previous run's finalize and still open across this one loses its base snapshot and fails its commit; only overlapping runs on one store can hit that, and K = 0 never expires). The expiry squashes every commit before that cutoff — the oldest retained run's own intermediate commits included — into its finalize snapshot; the newer runs keep theirs until a later cutoff passes them. Leaf objects are never touched by any of it.

output.icechunk: false opts a hive run out (default on; excluded from the D19 semantic core like sweep). Windowed (morton-hive/2) leaves and raster hive products are not indexed yet (spec §11.6: the row model holds them, the writer does not record them, and the knob resolves off there; issue #584 tracks both); the sweep's overviews are in — every declared overview level has its group in the store's one repo.

Reading it back:

import icechunk, zarr

# The virtual chunk container's prefix: the store root URL with a trailing
# "/", recorded in the repo's own root attrs as zagg_icechunk.url_prefix.
root_prefix = "s3://bucket/product/"
storage = icechunk.s3_storage(
    bucket="bucket", prefix="product/icechunk", region="us-west-2", anonymous=True
)
repo = icechunk.Repository.open(
    storage,
    authorize_virtual_chunk_access={
        root_prefix: icechunk.Credentials.S3(icechunk.s3_anonymous_credentials())
    },
)
group = zarr.open_group(repo.readonly_session("main").store, mode="r")
group.attrs["multiscales"]         # the manifest's zagg-multiscales/1 block: every level
rows = group.attrs["zagg_icechunk"]["rows"]   # the row labels, in allocation order: ["all"] when unwindowed
row = rows.index("all")            # look a row up by label; never assume an order
count = group["19/count"]          # the base leaves: shape (n_rows, 12·4^19), chunks (1, 4^6)
column = group["13/count"]         # the leaf columns' member: shape (n_rows, 12·4^13), chunks (1, 256) (one per leaf)
coarse = group["11/count"]         # the order-7 overviews: shape (n_rows, 12·4^11), chunks (1, 256) (one per node)
cells = count[row, 0 : 4**10]      # one leaf's cells of that row
group["window_start"][:], group["window_end"][:]   # each row's [start, end); the fill value on "all"

# A run's snapshot, by tag (spec §11.4): the store as that run left it.
as_of_run = zarr.open_group(repo.readonly_session(tag="run-<run_id>").store, mode="r")
runs = [s.metadata for s in repo.ancestry(branch="main") if s.message.startswith("finalize ")]

That is the public store — the motivating case, and the one an icechunk-js reader in a browser takes. A private store swaps anonymous=True for from_env=True and s3_anonymous_credentials() for s3_from_env_credentials(), both of which resolve the AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY environment variables only — not the botocore chain, so an AWS_PROFILE reader wants get_credentials= and icechunk.s3_refreshable_credentials(...) instead (which is what the writer itself does, icechunk_refs._boto3_credentials). That callable must hand icechunk an expires_after whose tzinfo is datetime.timezone.utc itself; botocore's refreshable expiry is a dateutil tzinfo (tzutc() or a tzoffset), which icechunk rejects, so convert it with .astimezone(datetime.timezone.utc) as _boto3_credentials does. Pass region= either way: s3_storage leaves it to a guess otherwise, and zagg's stores are us-west-2. Nothing here imports zagg — the repo opens from its own recorded url_prefix and the paths above, which is what makes the same two calls writable in icechunk-js.

The manifest split (spec §11.5) cuts both axes: the cell axis at one base manifest per order-split_order cell — at the defaults an order-6 cell, 16,384 chunks, 64 leaves — and the same number of chunks per manifest at every coarser level (so a coarse manifest spans a coarser cell: order 2 at /13, a base cell at /11 and above); and the row axis at rows_per_manifest rows, one row per manifest by default, so an append adds manifest files and rewrites none of the rows already written. Both are recorded in the repo root's zagg_icechunk block (split_order, rows_per_manifest) and per order group as levels.{order}.split (chunks, order, rows), so a reader never assumes them.

Raster hive stores (issue #247)

Raster (pull-NN) pipelines write the same tree with windowed (time, cells) leaves: one vanilla zarr v3 leaf per (shard, window) unit at shard_leaf_path(root, shard, window=label), each carrying leaf-local time (int64 microseconds with CF attrs by default, or uint64 mortie toc words carrying the spec §8.1 temporal declaration at shape: "coordinate" and no CF attrs under output.time_encoding: toc — which the shipped Sentinel-2 config sets, issue

443) and morton (packed u64 words) as the sole cell

coordinate — stored on an unwindowed leaf, derived on a windowed one, which stores no such array (Time windows); cell_ids (NESTED) rides only the emit_cell_ids transition hatch (issue #304) — plus one (T_leaf, cells_per_shard) array per configured band, chunked (1, cells_per_chunk). The leaf's time axis is the unit's own acquisition groups — known at dispatch from the catalog, so both dispatchers produce identical leaves — and its coords are written at template time (nothing is deferred to a per-shard coords pass). There is no flat global template on the hive branch: template time writes only morton_hive.json (D5/D6).

Differences from the aggregation path, all espg-ratified on issue #247:

• Window membership is the acquisition's STAC datetime, decided at dispatch — there is no per-observation timestamp column, so no observation-level filter is injected. output.windowing.time_field is optional (fixed to datetime, which the manifest temporal block records); the epoch/scale/units conversion knobs are rejected (STAC datetimes are already ISO-8601 UTC). An acquisition group (entries sharing a time_key — one datatake's adjacent MGRS tiles) belongs to the window containing its earliest datetime within the shard, so a group never splits across leaves at a boundary.
• Schedule none is supported for consistency (ratified): one bare {full_id}.zarr leaf per shard carrying the full time axis; re-run = whole-leaf replacement; D14 "full" gated off exactly as aggregation gates it. The append cost (a re-run rewrites the whole leaf) is the user's explicit choice, visible in the manifest.
• Coverage is popcount-decided per D14 from the spatial union of the unit's acquisitions (per-timestep validity stays data-plane nodata, D9): an interior shard — every child cell covered — stamps encoding: "full" with no sidecar PUT; an edge-of-scene/swath shard writes the real bitmap sidecar.
• The D15 stamp truth is the window label plus the actual ISO-UTC [min, max] of the unit's acquisition datetimes and the acquisition count; the root coverage.moc unions the per-leaf ranges as cache.
• sharded: true is permanently excluded on the raster path (not deferred): per-timestep slab streaming would read-modify-write each ShardingCodec object once per timestep, and raster object count is time-axis-dominated anyway.

The shared worker (zagg.processing.raster.process_and_write_raster_hive, the raster analog of process_and_write_hive) runs identically under the local dispatcher and the Lambda mode: "process_raster" hive branch; hive events carry no time_index (the leaf axis is unit-local) plus an optional window, while flat raster events stay byte-identical to pre-#247 runs. The manifest rides the same ping → async-setup → finalize-backstop lifecycle as aggregation (issue #252).

Reading a hive store

There is no store-root zarr.open() (deliberately — D12; a root hierarchy can be added later by the sweep as a derived artifact). Readers:

1. GET morton_hive.json (once, cacheable) → shard_order, cell_order.
2. GET coverage.moc (zagg.coverage.load_coverage) → the covered shard set, intersected with the AOI (root_coverage_and) — see Coverage.
3. Compute a shard's leaf path by string arithmetic on its decimal id (zagg.hive.shard_leaf_path), open the leaf zarr, and check the commit stamp (zagg.hive.read_commit) before trusting the contents; the stamp's coverage payload pre-filters the AOI (box_and/bitmap_and). If the stamp names current (a versioned leaf), the arrays are under {leaf}/{current}/ (zagg.hive.resolve_leaf); the stamp's first GET is the pointer, so following it costs no extra request for a reader that already GETs the root stamp and then addresses chunks by key (moczarr's 2-GET path, zagg's readers). A zarr-python reader that opens the version group pays one GET for its {current}/zarr.json. A windowed leaf has no {cell_order}/morton array to open: the cell words are the shard's children at the cell order (zagg.grids.morton.cell_words(shard, cell_order)), the shard being the leaf id the path was computed from.
4. Discovery without a root MOC falls back to the delimiter-LIST walk: recurse on [1-4]/ children; a *.zarr entry is data at that node; no digit children ⇒ nothing finer. Never LIST per observation in a join loop (D10).

The store-root coverage.moc (issue #200 phase 3, default-on for hive) removes the walk from the bootstrap path: one GET of the root object yields the shard-order coverage MOC (JSON ranges, decimal-string endpoints). It is written fail-open at end of run — by the dispatcher directly (local) or one fire-and-forget mode: "coverage" worker invoke (Lambda; an older deployment has no coverage mode and 400s the event in its process handler — logged, no writes, no async retry — which is safe: the object is a regenerable cache under D9, and readers degrade to the walk). Incremental runs union with the existing object; concurrent runs race benignly (GET-union-PUT is not atomic: last writer wins, and its union may miss the loser's shards until the sweep or the next run re-unions — accepted under D9/O7). The §7 sweep remains the authoritative rebuilder.

The live stores (issue #560 cutover, 2026-10-08)

The ATL03 demo store was rebuilt from scratch on the 0.57.0 fleet as atl03_tdigest_o9_v3.zarr and is the store every ATL03 reader in this tree now names (issue #560; the runbook is the operator's demo/15_rebuild_2026-10.ipynb, kept outside the tree). It was built from the packaged atl03_tdigest_strata_healpix template (plus output.pyramid: {overviews: 13}; the template then shipped pyramid: false, and since issue #620 declares the v4 ladder {overviews: [16, 15, 14, 13]} itself — see the templates declare the ladder) against the frozen operator-local shardmaps shardmap_california_o9.json (2,726 shards) and shardmap_neon_aop_o9.json (251 shards, 13 overlapping), not in the tree, so its manifest is born at the post-epoch digest aacfe1e387d2… (no index-exclusion migration applies) and carries: cell 19 / shard 9 / chunk_inner 13; the dense zagg-pyramid/2 ladder 13..4 declared at init together with its multiscales mirror; versioned leaves (pointer roots, spec §1.5); four-field leaf columns from birth; and an Icechunk companion repo at …/atl03_tdigest_o9_v3.zarr/icechunk (zagg-icechunk/2, rows ["all"], tagged run-9f01f911935a4b6184e3ac87ec843895 for the CA run and run-9a12b233146049f399ae34cc66e7ec2f for the NEON append). Its manifest is vendored as tests/data/ca_atl03_tdigest_o9_v3_morton_hive.json and is the anchor tests/test_live_store_templates.py pins the template to. Two gaps were known at cutover: the root temporal section is short (the run tail's families sweep dies at the 900 s wall on a store-scale run, issue #610), and five NEON shards OOMed on the 4 GB tier and await a re-dispatch (coverage was 2,959 shards; the ladder sweep was still running). atl03_tdigest_o9_v2.zarr was deleted on 2026-10-07 and is referenced nowhere. All stores below sit under s3://us-west-2.opendata.source.coop/englacial/zagg/demo/ and are anonymously readable.

store	role	built	frozen semantic_hash
atl03_tdigest_o9_v3.zarr	live ATL03	0.57.0 fleet, 2026-10-08	aacfe1e387d2… (post-epoch)
gedi_flux_o9.zarr	live GEDI — unchanged, no GEDI rebuild this round	0.52.0, 2026-08	4f8287947a83… (pre-epoch)
atl03_tdigest_o9.zarr	v1 ATL03, retained as the comparison store	2026-08-19	b9b15fdde78f… (pre-epoch)
serc_atl03_v3.zarr	current SERC canary (4 shards; ladder swept, pyramid_check PASS, tagged run-e10d7d83f79d46d8913cd41d88ff54c3)	0.57.0	—
serc_atl03_v2.zarr	the 0.55.0 fleet's SERC canary	0.55.0	—

Status

• Both backends write hive stores end-to-end through the same zagg.hive.process_and_write_hive code path. On Lambda (issue #199 phase 3) the manifest write fires as an async mode: "setup" Event invoke at init, with mode: "finalize" as its idempotent backstop (issue #252 hybrid; a lightweight mode: "ping" preflight keeps the pre-fan-out fail-fast) — the orchestrator still needs no S3 access — and each worker derives its leaf path from its shard_key + the event config's orders, emits its own leaf template, and stamps completion as its final PUT. The async status channel stays at the flat sibling prefix ({store_root}.status/<run_id>/…), outside the digit tree.
• Coverage ships (issue #200 phases 1–4): the tier-0 morton box on the commit stamp, the exact zstd-bitmap coverage.moc sidecar inside each leaf, the end-of-run store-root coverage.moc (shard-order ranges MOC, output.coverage_moc, default on for hive) for the one-GET bootstrap, plus the zagg.coverage reader primitives (per-tier AOI intersection, O7 staleness lean, explicit refresh).
• Dense arrays shard inside the leaf (issue #236): hive output is byte-identical to the flat sharded layout — one ShardingCodec object per dense array per leaf (plus one per ragged field, issue #209), the default at K > 1, so a leaf costs one PUT per dense array instead of K per-inner-chunk PUTs concentrated on a single prefix.
• Skip-if-current re-runs ship for hive point runs on every backend (issue #388; the fleet arming landed with PR #581): the worker-side identity gate, the contraction guard, and the lifecycle touch — see Re-runs. Raster fleet units are not armed yet.
• Write-throughput validation at fleet scale is tracked with the benchmark machinery in issue #202.