Scarf single-cell analysis

This skill encodes how to drive core Scarf for single-cell RNA analysis: which calls to make, in what order, what evidence to check, and what not to do. Scarf streams Zarr-backed count matrices in bounded blocks and records every computation as an immutable, content-addressed artifact, so an identical call returns the existing result instead of recomputing it. The skill covers the whole path: converting H5AD, 10x, MTX or Seurat data into a Scarf store, QC with removal audits, highly variable genes, PCA and neighbour graphs, Leiden and Paris clustering, UMAP, markers and cautious annotation, batch correction, donor-level comparisons, headless plotting, provenance and export. It does not use or describe the automated scarf.agent package.

The core mental model is simple. A DataStore is one Zarr directory with a shared cell table, one group per assay, artifacts and pipeline runs. Every result is an artifact reference, and producers take references and return references. A single pipeline run executes the whole RNA recipe (filtering, cell cycle, HVG, normalization, PCA, graph, UMAP, Leiden at several resolutions with a silhouette pick, Paris, doublet scores, markers) and returns a durable run object with frozen cell and feature views and a report.

Use it when a task involves a Scarf .zarr store, scarf.DataStore, ds.pipeline or a PipelineRun. Use it when converting H5AD, 10x, MTX or Seurat data into a Scarf store, or when opening and mounting a Cytebase dataset. Use it for single-cell RNA-seq analysis with core Scarf, and when counts are too large for memory or live on object storage and must be streamed under an explicit memory budget.

Do not use it for scarf.agent, which this skill does not cover. It also needs Python 3.12+ and scarf 1.0.0rc17 or newer, a pre-release; a bare install resolves to the old 0.32 series, whose API differs. Cluster labels produced this way are hypotheses backed by marker evidence, not a validated cell-type classifier.

Input is a Scarf store, or raw data to convert into one. Resources are set per process through SCARF_MEM_BUDGET and SCARF_WORKERS, because defaults claim all detected RAM and every CPU. For headless runs, set MPLBACKEND=Agg, call plots with show=False, then save and close the result.

Outputs are kept in an output directory so another person or agent can audit and continue: an analysis_log.md with one entry per step (decision, alternatives, evidence with numbers, figure paths), a handoff.json with the run label, run ID and the references relied on, a run report and a lineage file, figures as PNG and tables as CSV (markers, cluster sizes, composition, label map).

The skill walks through nine steps: inspect, design, QC, baseline run, structure, branch if needed, annotate, compare, report. Each step names the evidence to record and the reference module to read. The first look at a store is read-only, checking cell and feature counts, runs, artifact kinds and whether counts look raw, then reviewing every metadata column for design columns, held-out author annotations and author-derived columns.

Key rules it enforces include: never filter by editing the live cell key; keep every reusable reference in a dict and never guess artifact IDs; audit QC removals per sample and per marker-defined group; use a new label for every run variant; read pipeline outputs instead of recomputing; align arrays through the cell selection and join on cell ids, never row order; budget count passes on mounts; run one DataStore per process; pass an explicit HVG blacklist; name a cluster only with two or more specific positive markers plus low lineage-negative markers and a plausible QC and doublet profile; and treat cells as non-replicates, so condition claims need donor-level aggregation with at least two independent donors per group. Annotation columns that will judge the result are held out and used only in a final, clearly separated comparison, reporting ARI and per-type F1.

The skill also ships a troubleshooting table that maps symptoms such as a read-only open error, fewer active cells than expected, a reused run label, or very slow steps on a mount to their likely cause and fix. Long steps are written as scripts that log to a file and end with a DONE or FAILED line, run in the background and waited on with a bounded loop.

It works with core Scarf and its Zarr stores, reads H5AD, 10x, MTX and Seurat data, can mount Cytebase datasets (which need the cytebase extra and network access), and exports AnnData, H5AD, MTX, CSV and pseudobulk tables. Plotting is headless through matplotlib. The catalogue copy is Markdown only; the maintained copy in the Scarf repository also ships a read-only store inspection script.

Setup, taken from the skill:

pip install "scarf[extra]>=1.0.0rc17"

Bioinformaticians and analysts who run single-cell RNA workflows in Scarf and want a disciplined, auditable routine: reproducible runs, explicit QC decisions, honest annotation, and no overclaiming from cells that are not independent replicates. It suits people working with datasets too large for memory or stored remotely, and teams that need a handoff record another analyst can resume from.