Explore Hugging Face Datasets via API
Read-only exploration and querying of Hugging Face datasets via the Dataset Viewer API.
Why it matters
Access and explore Hugging Face datasets programmatically using the Dataset Viewer API. This skill enables read-only exploration, data extraction, and querying of datasets hosted on Hugging Face.
Outcomes
What it gets done
Validate dataset availability and retrieve metadata.
Paginate, search, and filter dataset rows.
Retrieve parquet links for advanced querying.
Upload and manage datasets via CLI.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/ag-hugging-face-dataset-viewer | bash Overview
Hugging Face Dataset Viewer
Provides read-only exploration of a Hugging Face dataset through the Dataset Viewer API: validating, previewing, paginating, searching, filtering, and SQL-querying parquet shards. Use it for read-only exploration and extraction of a Hugging Face dataset, keeping all Dataset Viewer calls side-effect free.
What it does
Provides read-only exploration of a Hugging Face dataset through the Dataset Viewer API (base URL datasets-server.huggingface.co): validate availability (/is-valid), resolve config and split (/splits), preview (/first-rows), paginate rows (/rows with offset/length, max 100), full-text search (/search) and predicate filtering (/filter), and retrieve parquet shard links (/parquet) plus size and column statistics (/size, /statistics) and Croissant metadata (/croissant) when available. Gated or private datasets require an Authorization: Bearer <HF_TOKEN> header.
curl "https://datasets-server.huggingface.co/rows?dataset=stanfordnlp/imdb&config=plain_text&split=train&offset=0&length=100"
For SQL-style querying, it derives Hub parquet alias paths (hf://datasets/<namespace>/<repo>@~parquet/<config>/<split>/<shard>.parquet) from the /parquet endpoint and runs them through npx parquetlens, including COPY ... TO exports to CSV, JSON, or Parquet. For creating and uploading datasets, it offers a zero-dependency flow - create the repo and upload parquet files via the Hub UI, then verify shards via /parquet - or a low-dependency CLI flow (npx @huggingface/hub upload, with a --private flag for a private repo on creation), reading the auth token from HF_TOKEN.
When to use - and when NOT to
Use it for read-only exploration and extraction of a Hugging Face dataset - validating, previewing, paginating, searching, filtering, or SQL-querying its rows - keeping all Dataset Viewer calls side-effect free.
Inputs and outputs
Input: a dataset namespace/repo identifier, plus config, split, and pagination or filter parameters. Output: JSON rows, search or filter matches, parquet shard links, size/statistics metadata, or a SQL query result exported to CSV, JSON, or Parquet.
Integrations
Runs against the Dataset Viewer API (datasets-server.huggingface.co) and the Hugging Face Hub (huggingface.co/new-dataset, the @huggingface/hub CLI), with parquetlens for local SQL querying of Hub-hosted parquet files.
Who it's for
Developers exploring, querying, or uploading Hugging Face datasets who want direct API and CLI workflows instead of downloading full datasets locally.
Its core workflow is sequential: optionally validate the dataset with /is-valid, resolve the config and split with /splits, preview with /first-rows, then paginate with /rows before reaching for /search or /filter. When pagination is partial, response fields like num_rows_total, num_rows_per_page, and partial drive continuation logic, and /search matches string columns while /filter requires predicate syntax in a where parameter with optional sorting via orderby - both calls stay strictly read-only and side-effect free. Query parameters must be URL-encoded, offset is 0-based, and the API's default HTTP method throughout is a plain GET request, with a length cap of 100 rows per page on row-like endpoints.
Uploading a parquet folder via the CLI auto-creates the destination dataset repo if it doesn't already exist, and after any upload, calling /parquet again discovers the resulting config/split/shard values needed for querying that same data back through the @~parquet alias.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.