Weather Model Run Discovery

Weather Model Run Discovery resolves a numerical weather prediction run before any model fields are downloaded: it probes exact objects or narrow prefixes, verifies a request-specific completion contract, and returns the newest qualifying initialization with its source metadata. It decides which run and objects exist - a separate fetching skill handles the actual transfer or subsetting afterward.

Its central discipline is defining "complete" before probing: the model, domain, product, and ensemble member; legal UTC cycle hours; required forecast hours; required companion objects (pressure and surface products together, for example); required inventory sidecars; and provider fallback rules. A run counts as complete only when every sentinel the consumer actually needs exists - the presence of forecast hour zero alone does not prove a run is complete, since later forecast hours can still be publishing.

aws s3api head-object --no-sign-request \
  --bucket BUCKET \
  --key "EXACT/DOCUMENTED/OBJECT"

aws s3api list-objects-v2 --no-sign-request \
  --bucket BUCKET \
  --prefix "MODEL.DATE/DOMAIN/PRECISE-RUN-PREFIX" \
  --max-items 100

The discovery workflow normalizes all candidate initialization times to UTC, enumerates only legal cycles newest-first within a lookback bound, builds documented exact object keys rather than recursively scanning a bucket, and probes metadata only (HeadObject, HTTP HEAD, or a narrow paginated listing) rather than the model payload itself. It rejects zero-length or placeholder objects, and for a live mirror that may expose objects before publication finishes, it repeats the metadata probe after a bounded interval and requires stable size and object identity before accepting a candidate. A fallback provider is only accepted if it refers to the exact same initialization, product, member, and forecast hour - never a different one mixed in silently.

A minimal resolver pattern is given that separates provider-specific key construction from the selection rule, accepting a metadata-only probe callback so the same logic works against S3, HTTP, or a test fixture, and raising a clear error listing every cycle checked and which sentinels were missing when nothing qualifies. The resolution record it emits captures requested time, model/product/initialization, required forecast hours, provider, and per-object role/URI/size/ETag/last-modified - with ETags explicitly treated as opaque identity values rather than assumed to be full-file MD5 checksums.

Reliability guidance covers caching positive results only briefly (matched to model cadence and publication latency) and negative probes for even shorter, retrying transient errors with bounded exponential backoff and jitter while honoring Retry-After, and bounding total lookback and probe count so a provider outage can't trigger an unbounded bucket walk. It explicitly warns against treating one mirror's lag as proof a run is absent everywhere, and against mixing objects from different providers without verifying their run identity and product semantics are actually equivalent.

Use it when an application needs the newest usable GFS, GEFS, HRRR, RAP, or NAM cycle; when the nominal latest cycle 404s or has only early forecast hours; when equivalent AWS, NOMADS, Google, or Azure mirrors need comparing without silently mixing runs; when a scheduled job needs a bounded fallback to an older complete cycle; or when the resolved run and provider must be recorded for later reproduction.

Do not use it when the initialization is already fixed, or when the actual task is downloading, decoding, or interpreting model variables rather than discovering which run to use.

Input is a completion contract (model, product, member, forecast hours, required sidecars, provider priority, lookback bound). Output is a resolution record naming the exact resolved initialization, provider, and per-object identity metadata (URI, size, ETag, last-modified) for every required object, plus a log of rejected candidate cycles and their missing sentinels.

Hands its resolution record to weather-model-data-fetching for the actual data transfer; probes public NOAA Open Data S3 buckets via unsigned aws s3api calls.

Weather-data engineers building pipelines that need the newest genuinely complete NWP model cycle - avoiding the common failure of grabbing a run whose forecast-hour-zero file exists but whose later, actually-needed hours are still publishing.