Typed morton boundary¶
zagg's morton output coordinate is a real datatype, not an anonymous integer
column. mortie defines the morton_index type once, and zagg carries it typed
across every surface it crosses (issue
#135):
| Surface | Carrier | Form |
|---|---|---|
| pandas carrier (worker) | pandas.DataFrame |
mortie.MortonIndexArray extension array (#71) |
| arro3 carrier (worker) | arro3.core.Table |
morton_index Arrow extension type over the PyCapsule C Data Interface (added in mortie 0.8.4) |
| catalog / shardmap parquet | pyarrow.Table |
morton_index pyarrow extension type (registered by mortie on import) |
| Zarr store (on disk) | Zarr v3 array | plain uint64 — the packed words, byte-for-byte |
Who owns what¶
mortie owns the interchange type; zagg stores uint64. The extension
metadata (ARROW:extension:name == "mortie.morton_index") lives at the
interchange layer only. The write boundary
(zagg.processing.write._iter_carrier_columns) extracts the packed uint64
words from a typed column — on either carrier — so the on-disk output is
identical to a plain-uint64 write. Reads reconstruct the typed array via
zagg.grids.morton.to_morton_array.
The adapter lives in zagg.grids.morton:
morton_to_arrow(values)—MortonIndexArray(or raw words) →arro3.core.Arraycarrying the extension type, zero-copy via__arrow_c_array__;morton_from_arrow(col)— any Arrow C-Data source (array, chunked column, or capsule pair) →MortonIndexArray;is_morton_arrow(col)— detects the extension type from field metadata.
No pyarrow is needed on any worker path: the arro3 carrier and mortie's
PyCapsule surface speak the Arrow C Data Interface directly (pyarrow stays in
the off-Lambda catalog extra). Arrow nulls map to mortie's all-zero empty
sentinel word in both directions, so missing values round-trip.
cell_ids: morton-only storage (D16)¶
Since the issue #304 flip, morton is the only stored cell coordinate:
the packed uint64 words carry the cell identity (and its order,
intrinsically), the store's dggs attrs declare grid name: "morton" +
coordinate: "morton", and NESTED HEALPix ids are derived at read (the
moczarr fabrication layer). The issue #135 cell_ids_encoding knob is
retired — a config still carrying it fails validation with a migration
pointer.
For transition stores (browser-direct demos until the gridlook morton decode
lands), the escape hatch keeps writing the legacy NESTED array in addition
to morton:
output:
grid:
type: healpix
parent_order: 6
child_order: 12
emit_cell_ids: true # default: false — morton only
A hatch store's declared coordinate stays morton; cell_ids is an extra
legacy member. The grid's spatial_signature() (recorded in ShardMap
manifests) is unchanged by the hatch — deliberately, so shard maps remain
reusable — while the full signature() records emit_cell_ids (the hatch
changes the leaf schema, so mixed states never co-aggregate). (The
pre-existing output.grid.indexing_scheme config key is descriptive only and
must stay nested.)
Shardmap parquet manifests¶
ShardMap.to_parquet / ShardMap.from_parquet are the Arrow-native siblings
of the JSON manifest: shard_keys is a typed morton_index column, granule
and AOI payloads ride as per-shard JSON strings, and provenance lives in the
schema metadata (mirroring the Catalog geoparquet convention). This path
requires pyarrow (pip install zagg[catalog]) and runs off-worker.