Upgrading a published store from `/1` to `/2`¶

A store built before its aggregation fields were declared composable has no leaf columns: the gen-1 partials the zagg-pyramid/2 staged sweep folds are written by the leaf worker at aggregation time, and only when the writing run's declaration admits the fields. Such a store cannot take the /2 staged sweep at any transport.

This page is the runbook that upgrades one without re-aggregation. The cost is one leaf-reading pass — the same order as one /1 cascade build, paid once — because every input the column fold ever had is still sitting in the leaf.

Preconditions¶

The store is quiesced. No aggregation run may be writing the same (leaf, window) while the backfill runs. The backfill is the one sweep-side writer of a leaf artifact, so it is the one pass for which the spec's fleet ∥ sweep disjointness does not hold (§4.6). The sweep-admission lease (§4.8) serializes it against other sweeps; nothing can serialize it against a fleet — proving quiescence would need a global run registry zagg deliberately does not have.

As a smoke alarm the pass lists the store's status channel ({store}.status/run-*/, the per-run objects the event transport writes) and logs a loud WARNING naming any run with activity in the last 900 s — one Lambda worker wall, so an older object cannot belong to a unit still running. It never refuses and never fails the pass, and it is fail-open: a status channel that is absent, unreadable, or scoped away is silent. Silence is not proof. A store written by a pre-#327 dispatcher, or one whose status prefix was lifecycled away, looks exactly like a quiesced one — the precondition is still yours to confirm. - You have the config the store was built with. declare_pyramid refuses a config whose semantic_hash the store's frozen one denies. output.pyramid is not in the semantic core, so adding or changing it on the original config hashes identically (the leaf-shaping output keys — aoi_mask, windowing, time_source, grid.sharded — do move the hash). For the two live demo stores the originals are the packaged templates (atl03_tdigest_strata_healpix, gedi01b_waveform_healpix_hive) with count declared int32 — the templates now declare it int64 (#626), a hashed key, so a config built from the templates as shipped is refused against those stores until they are rebuilt. Use each store's run-record config (vendored for the v1 ATL03 store and gedi_flux_o9 as tests/data/*.config.json); tests/test_live_store_templates.py pins both digests and proves count is the only divergence for GEDI, and count plus the yearly output.windowing block — a hashed D13 key, since issue #638 — the only two for ATL03 (issue #547). The templates also declare the leaf ladder now — output.pyramid: {overviews: [16, 15, 14, 13]} since issue #620, where the live stores were built with pyramid: false (GEDI) or {overviews: 13} (the v3 ATL03 store) — and pyramid is hash-invisible, so the semantic guard does not catch that difference: declare_pyramid derives the block from the config it is handed and replaces a differing declaration (previous: replaced). Hand it the run-record config, or the template with output.pyramid set to what the store declares, unless re-declaring the ladder is the intent. A store built before the index-exclusion epoch (issue #499 — data_source.index left the semantic core) is accepted on the same terms, and step 1 is also its migration: declare_pyramid recognizes the config's pre-epoch digest and rewrites the frozen semantic_hash in the same manifest write. The tools/redeclare_dense_ladder.py dry run says so before anything is written — read for the line semantic guard: legacy MATCH (b9b15f…) → will rewrite to aacfe1… (a plain MATCH means no migration; a MISMATCH means --execute will refuse). Until that write has happened the store refuses appends by name — the append path never migrates on its own — so run this step before any post-epoch aggregation run into the store. - The store's leaves are committed and its run records are at the product root (they are what discovery reads — never a recursive LIST).

The four steps¶

An append to the store mid-upgrade needs no sweep: workaround. A hive config that declares a ladder chains the staged sweep in its run tail by default (issue #620), and until step 1 has run this store's manifest declares none — the two disagree, which is the state this page exists for. The fleet dispatcher reads the store's own declaration once before it fans out ([issue

635](englacial/zagg#635)), so such a run **fires¶

nothing*: one manifest GET, one warning naming --declare-pyramid, zero invokes, where it used to bill one dying invoke per dispatch node. The run's stats/moc/submap rollups ride that sweep's node closes, so they are not folded either — the run warns and python -m zagg.sweep <store> --families stats,moc,submap is the backstop. The leaf side is the half that still needs care: it follows the *config's ladder, not the manifest, so append with output.pyramid set to what the store declares until the upgrade is done (hive_layout.md).

1. Re-declare the manifest to `zagg-pyramid/2`¶

python -m zagg.sweep s3://bucket/prefix/product \
    --declare-pyramid config.yaml \
    --overviews 13

--overviews takes the leaf cell resolutions, finest first, and is the lever this upgrade needs: without it the grid-less retrofit path can only declare /1, because the issue #384 default flip fires off the grid's resolved chunk_order and declare_pyramid has no grid to ask. The resolutions are validated against the manifest's own shard_order and cell_order — the store's truth wins over the config's — and nothing is written if they do not fit.

The API form takes either lever:

from zagg.sweep_overview import declare_pyramid

declare_pyramid(root, config, overviews=13)      # spell the schedule outright
declare_pyramid(root, config, chunk_order=13)    # or complete the #384 default

They are mutually exclusive (chunk_order= is inert once a schedule is spelled, so passing both refuses rather than silently ignoring one), and chunk_order= refuses a config that declares output.pyramid: false — there is no pyramid for it to default. overviews= does override a pyramid: false config, loudly: that is the point of the lever on a store whose build declared no pyramid.

Check the summary before moving on. declared_via names the lever, overviews is the expanded (node, cells) ladder the manifest now records, and fields is the per-field D24 class map — if every entry there is none, stop: the backfill will refuse, and the fix is the classifiers in the config, not the store.

2. Backfill the leaf columns¶

python -m zagg.sweep s3://bucket/prefix/product --families columns

columns is a registered sweep family, so it needs no new transport and no new mode: the work-set normalization and discovery (the same run-record scan, never a recursive LIST) are inherited whole. It is deliberately not in the default family set — a backfill is an explicit upgrade of a quiesced store, never a routine rollup's side effect, so it must be spelled.

Three entry points, all the same pass. This CLI, run_sweep(root, leaves, families=["columns"]), and a fleet mode: "sweep" invoke naming the family — the handler's sweep arm forwards the event's families and partition blocks as of #528. A fleet-scale partitioned backfill is the one combination still gated: the runner fires partitions as concurrent Event invokes and the sweep lease is store-granular, so it admits exactly one of them. Until that is ruled, fan a fleet backfill out as a single unpartitioned invoke, or partition it sequentially in-process.

--partitions 2^n bounds peak memory, but is not free here. Unlike --stages, which sweeps every partition under ONE lease, --partitions gives this family one run_sweep per partition and the lease is acquired and released per partition — so a --partitions 64 pass leaves 63 unleased windows in the middle of the very pass §4.6 sanctions as the second writer of a leaf artifact. On a genuinely quiesced store (the precondition above) that is harmless, since nothing else is admitted anyway; if you cannot guarantee quiescence, run the backfill unpartitioned so the whole pass sits under a single lease.

Per (leaf, window) the pass reads the leaf's commit stamp, skips the leaf if its column is already current, and otherwise recomputes the column from the leaf's stored arrays and writes it wholesale under the leaf's node prefix. The summary counts written / current / empty (no committed leaf) / failed. It is idempotent: a second pass writes nothing.

failed must be 0 before step 3. A leaf that cannot be read or folded is counted and skipped rather than aborting the pass, so a partial result comes back as an ordinary summary (with a logger.error) — and the staged sweep would then fold a ladder over whatever columns happened to land, recording the gaps only as source_children under-coverage in artifacts nobody reads. Check the count, fix the leaves it names in the log, and re-run the backfill; it is idempotent, so the second pass touches only what is missing. A pass that manages to write NOTHING at all does not return a summary: it raises, naming the last error, because every leaf failing the same way is a store-wide fault (expired credentials, a denied column prefix, an outage), not N leaf faults.

The gate is the manifest's declaration, and every refusal names itself:

Manifest says	What happens
`zagg-pyramid/1` (an `orders`/`spacing` schedule)	refused — re-declare (step 1)
`orders: []` (declared off)	refused — re-declare (step 1)
`/2`, but every field `class: "none"`	refused — fix the classifiers, re-declare
`/2` with composable fields	backfilled

3. Build the ladder with the staged sweep¶

python -m zagg.sweep s3://bucket/prefix/product --stages --scope all

Nothing about this step is upgrade-specific — it is the ordinary /2 staged sweep, and it now finds the columns it needs. Only run it once step 2 reported failed: 0: this step reads the columns, and it cannot tell a leaf whose column the backfill never wrote from one whose declaration carries none. The resulting ladder is byte-identical to the one a store built pyramid-ON from the same inputs produces (the acceptance test in tests/test_column_backfill.py; the only differences anywhere are the sweep run's own run_id and the wall clocks).

4. Retire the `/1` ancestors¶

The /1 overview zarrs the old declaration left behind are now declared-off regenerable debris (D24 option A). declare_pyramid deliberately preserves the materialized actuals across the re-declaration, so the manifest still inventories what is on disk. Deleting them is optional and reversible — they are cache, never truth — and there is no automated arm for it yet.

What the upgrade does not change¶

No byte of the store format. A backfilled column is byte-identical to the one the leaf's worker would have written from the same leaf, the two provenance timestamps aside. zagg-column/1 is unchanged.
No leaf is rewritten. The backfill reads leaves and writes columns.
No re-aggregation. Nothing is re-read from the source granules.

Residual: the same-second skip key¶

A column records no generation block and no run_id for its source leaf, so the backfill's skip-if-current test keys on what the artifacts already carry: the recorded declaration (node/cell orders, group set, per-field provenance), the column's realized structure — every group's members and each member's dtype, fill value and attrs, compared against the template the run would write — then written_at ordering and the column stamp's copy of the leaf's granule_count. The structure term is what catches the two drifts the recorded zagg_column grammar cannot carry, both of which a re-declaration really can move: a companion channel added or dropped (location / temporal, neither recorded), and an exact field's dtype / fill_value. It also carries each member's chunk grid and codec chain, so the backfill is the pass that re-lays a fine column group (4^5 cells or more, spec §4.6 — orders 14–16 under overviews: [16, 15, 14, 13]) written whole by a worker predating the sharded layout (issue #631): the values are unchanged, the objects are rewritten on inner chunks, and pyramid_check's readback leg fails such a store by name until the pass has run. Both stamps resolve to whole seconds, so a leaf rewritten in the same second as its column, at an unchanged granule count, reads as current. On a quiesced store — the precondition above — that window is not reachable; backfill_columns(..., force=True) is the unconditional rewrite if you ever need it.

Upgrading a published store from /1 to /2¶