Quickstart

Zero-to-working setup for zagg.

Prerequisites

• Python >= 3.12
• uv package manager
• NASA Earthdata account (free)
• For Lambda deployment: AWS account, Docker (for building layers)

Installation

git clone https://github.com/englacial/zagg.git
cd zagg
uv sync --all-groups

NASA Earthdata Authentication

zagg reads ICESat-2 ATL06 data from NSIDC's S3 buckets in AWS us-west-2. This requires NASA Earthdata credentials. Three methods, in order of precedence:

1. Environment variables

export EARTHDATA_USERNAME=your_username
export EARTHDATA_PASSWORD=your_password

2. ~/.netrc file (recommended for persistent use)

machine urs.earthdata.nasa.gov
    login YOUR_USERNAME
    password YOUR_PASSWORD

3. Interactive prompt

The earthaccess library will prompt for credentials as a fallback if neither of the above is configured.

Note

These are read credentials for source data at NSIDC. Output store credentials (for writing results to S3) are separate AWS credentials -- see Writing to S3 below.

Building a Catalog

The catalog step queries NASA's CMR-STAC (public, no auth needed) and builds a shard-to-granule mapping for the grid in your config:

uv run python -m zagg.catalog --config atl06.yaml --short-name ATL06 --cycle 22 \
    --polygon antarctica.geojson

This produces a JSON file (e.g., shardmap_ATL06_2024-01-06_2024-04-07.json) that maps parent morton cells to the S3 URLs of HDF5 granules containing data for those cells. The processing step consumes this file.

To inspect the chunking interactively -- shard outlines, granule footprints, and a grid that appears on zoom -- use the shard-map viewer (pip install zagg[viz]). See the shard-map viewer notebook (archived), which runs on a synthetic example (no network needed) and includes manual in-browser verification instructions.

Local Processing

The simplest path -- no AWS Lambda needed:

# Process one cell, write to local Zarr:
uv run python -m zagg --config src/zagg/configs/atl06.yaml \
    --catalog catalog_ATL06_cycle22_order6.json \
    --store ./output.zarr --max-cells 1

# Dry run (shows what would happen, no processing):
uv run python -m zagg --config src/zagg/configs/atl06.yaml \
    --catalog catalog_ATL06_cycle22_order6.json --dry-run

Writing to S3 (Optional)

To write output to S3, set the store path to an S3 URI:

uv run python -m zagg --config src/zagg/configs/atl06.yaml \
    --catalog catalog_ATL06_cycle22_order6.json \
    --store s3://my-bucket/output.zarr

This requires AWS credentials configured via one of:

• ~/.aws/credentials
• Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
• An IAM role (e.g., on EC2 or Lambda)

Lambda Deployment (Optional)

For full-scale processing, zagg dispatches work to AWS Lambda. See Lambda Deployment for details. The short version:

1. AWS prerequisites: IAM role with S3 + Lambda permissions, S3 bucket for output
2. Build: deployment/aws/build_function.sh and deployment/aws/build_layer.sh
3. Deploy: deployment/aws/deploy.sh
4. Run:

   python deployment/aws/invoke_lambda.py --config atl06.yaml --catalog catalog.json
   

The function name defaults to process-shard but is configurable via the ZAGG_LAMBDA_FUNCTION_NAME environment variable.

Configuration

Pipeline behavior is defined in a YAML config with three required sections:

data_source:
  reader: h5coro
  groups: [gt1l, gt1r, gt2l, gt2r, gt3l, gt3r]
  coordinates:
    latitude: "/{group}/land_ice_segments/latitude"
    longitude: "/{group}/land_ice_segments/longitude"
  variables:
    h_li: "/{group}/land_ice_segments/h_li"

aggregation:
  coordinates:
    cell_ids:
      dtype: uint64
      fill_value: 0
  variables:
    count:
      function: len
      source: h_li
      dtype: int64
      fill_value: 0

output:
  grid:
    type: healpix
    indexing_scheme: nested
    child_order: 12

# Optional top-level fields:
catalog: catalog_ATL06_cycle22_order6.json
store: ./output.zarr
worker:            # lambda backend: pick a pre-provisioned worker size
  memory: 2048     # 2048 | 4096 | 8192 (see docs/deployment/lambda.md)
  extra_disk: false

HEALPix output writes a morton-hive, sharded store by default: one self-describing leaf zarr per shard under a morton digit tree, with the shard's inner chunks bundled into one object per array — see Hive store layout and Sharded storage. An explicit output.store_layout: flat (the single shared zarr store) remains for interop/debug but is deprecated — removal is tracked in issue #251. The default is grid-keyed, not pipeline-keyed: HEALPix raster output writes hive leaves too (issue #247); rectilinear grids keep the flat shared store.

Where the run writes, and whether it touches what it finds

output.store declares the destination, and output.touch declares what the skip path does to it:

output:
  store: s3://us-west-2.opendata.source.coop/englacial/zagg/demo/atl06.zarr
  touch: auto        # auto (default) | always | never

A run that skips a unit — everything already current — writes nothing for it. That is correct for the store but invisible to a bucket expiration rule, so zagg refreshes LastModified across the skipped unit's footprint (issue #388). S3 has no touch API, so the refresh is a server-side self-copy, and on a versioned bucket a self-copy writes a new full-size version rather than updating a timestamp in place.

Whether that is worth paying depends on whether the destination expires objects, which zagg cannot read cross-account (s3:GetLifecycleConfiguration is bucket-owner only) and which no bucket name encodes. So it is declarable:

value	behaviour
auto (default)	Touch, unless the destination is a known published bucket (zagg.store._PUBLISHED_BUCKETS, Source Cooperative today). Those are archival — no expiration rule to defeat — and versioned, so touching them only mints noncurrent versions (issue #495 phase 4).
always	Touch regardless of destination. For an un-negotiated external target whose expiry rule you know about but whose name says nothing.
never	Never touch, local paths included. For an archival destination, or any store where version churn outweighs the protection.

auto infers; the other two override the inference (issue #501). A path the policy excludes is reported as not applicable — touch_skipped_paths in the run record — never as a failure.

See src/zagg/configs/atl06.yaml for a complete example and the custom aggregations notebook (archived) for customization examples.

Reading the output

For products with per-cell t-digest fields, the zagg.readers package reconstructs dense, spatially faithful (64, 64, n_bins) tensors — with an occupancy mask and a shared (offset, gain) z-window per block (the reader contract) — or lossless raw value vectors from the stored digests. The t-digest reader notebook (archived) walks through read_tensors and read_raw_values end to end on a self-contained synthetic store.