Schema-Based Dataset Loading

Overview

Every dataset on the Mozilla Data Collective (MDC) platform has an associated schema.yaml file. This declarative file tells the SDK how to turn the raw files inside the archive into a ready-to-use pandas DataFrame, without executing any custom code outside the datacollective library.

from datacollective import load_dataset

df = load_dataset("your-dataset-id")
print(df.head())

Under the hood, load_dataset() performs the following steps automatically:

Resolve the schema: check local cache or the schema registry for schema.yaml. If the dataset is not registered this step raises a warning, so we never download an unsupported archive.
Download the archive (with resume support). The schema we fetched in step 1 tells the loader how the files are structured.
Extract the .tar.gz / .zip to a local directory.
Parse the YAML into a validated DatasetSchema (Pydantic model) and dispatch to the loader for the schema's root_strategy (index, glob, …), which returns the final DataFrame. When the schema declares a task with a known contract (ASR, TTS, LLM), the loaded DataFrame is checked for the task's required logical columns; a TaskValidationWarning is emitted if any are missing (the DataFrame is still returned).

The schema file describes:

How to find the data files (index file path, glob pattern, etc.).
How to map raw columns / files into a clean DataFrame.
Optionally, what task the dataset is for (ASR, TTS, …), which the loaded DataFrame is checked against (violations emit a warning).

Minimal example

dataset_id: "common-voice-gsw-24"
task: "ASR"
root_strategy: "index"
format: "tsv"
index_file: "train.tsv"
columns:
  audio_path:
    source_column: "path"
    dtype: "file_path"
  transcription:
    source_column: "sentence"
    dtype: "string"

This tells the SDK: "Read train.tsv as tab-separated, take the path column as audio file paths and the sentence column as transcriptions."

Schema fields reference

Required fields

Every schema must have:

Field	Type	Required	Description
`dataset_id`	`str`	✓	Unique dataset identifier on MDC.
`root_strategy`	`str`	✓	Loading strategy: `"index"`, `"multi_split"`, `"multi_sections"`, `"paired_glob"`, or `"glob"`. There is no default — every schema must set it explicitly.
`task`	`str`	✗	(optional) Task type as defined on the MDC Platform (`"ASR"`, `"TTS"`, …). When set to a task with a known contract, the loaded DataFrame is checked for the task's required logical columns (e.g. ASR/TTS: `audio_path` + `transcription`; LLM: `text`); missing columns emit a `TaskValidationWarning` (shown even with `enable_logging=False`) but the DataFrame is still returned. Tasks without a contract (e.g. `"OTH"`) load without validation.

Loading strategies

The remaining fields depend on which strategy the dataset uses. The strategy is selected with the required root_strategy field:

Strategy	When to use	Key fields
Index-based	A metadata file (CSV / TSV / pipe-delimited) lists each sample.	`root_strategy: "index"`, `index_file`, `columns`
Multi-split	Multiple split files (train, dev, test, …) each containing samples.	`root_strategy: "multi_split"`, `splits`
Paired-glob	Each audio file has a matching sidecar file (`.txt` for TTS, JSON for ASR), no index file at all.	`root_strategy: "paired_glob"`, `file_pattern`, `audio_extension` (TTS) / `format: "json"`, `record_path`, `columns` (ASR)
Multi-sections	Multiple section directories, each with its own index file.	`root_strategy: "multi_sections"`, `sections`, `section_root`, `index_file`
Glob	Directory-structured dataset with metadata encoded in the path hierarchy.	`root_strategy: "glob"`, `file_pattern`

Index-based fields

Field	Default	Required	Description
`root_strategy`	—	✓	Must be `"index"`.
`format`	Inferred from `index_file` when possible	✗	Optional format hint: `"csv"`, `"tsv"`, or `"pipe"`. Useful when the file extension is misleading.
`index_file`	—	✓	Path to the metadata file, relative to the dataset root.
`columns`	—	✓	Mapping of logical column names → source columns (see below).
`base_audio_path`	`""`	✗	Directory prefix or list of directories used to resolve `file_path` dtype columns. Entries may also use `${column}` placeholders from the current metadata row.
`separator`	Inferred from `format` or `index_file`	✗	Explicit column separator override (e.g. `"\\|"`).
`has_header`	`true`	✗	Whether the index file has a header row. When `false`, `source_column` must be a positional integer.
`encoding`	`"utf-8"`	✗	File encoding (e.g. `"utf-8-sig"` for files with a BOM).
`strict`	`false`	✗	Disable archive heuristics for deterministic loading: `index_file` must exist at its literal path relative to the dataset root (no recursive search) and source column names must match exactly (no fuzzy matching). Applies to every strategy that reads delimited files.

The index_file lookup is deterministic even without strict: the literal relative path wins when it exists; otherwise the tree is searched recursively and the shallowest match is used — multiple matches at the same depth raise an error instead of picking one arbitrarily.

Multi-split fields

Field	Default	Required	Description
`root_strategy`	—	✓	Must be `"multi_split"`.
`splits`	—	✓	List of split names to load (e.g. `["train", "dev", "test"]`).
`splits_file_pattern`	`"*/.tsv"`	✗	Glob pattern to locate split files.
`columns`	(optional)	✗	Column mappings applied to every split frame.
`base_audio_path`	`""`	✗	Directory prefix or list of directories used to resolve `file_path` dtype columns. Entries may also use `${column}` placeholders from the current metadata row.

Paired-glob fields

TTS (text sidecars) — each audio file has a matching .txt file with the transcription; pairing is done on the filename stem:

Field	Default	Required	Description
`root_strategy`	—	✓	Must be `"paired_glob"`.
`file_pattern`	—	✓	Glob pattern to find text files (e.g. `"*/.txt"`).
`audio_extension`	—	✓	Extension of the matching audio files (e.g. `".webm"`).
`columns`	—	✗	Optional mappings applied over the derived `audio_path` / `transcription` / `split` sources (rename, dtype, drop). The `split` column is kept. When omitted, the default `audio_path` / `transcription` / `split` output is returned.

ASR (JSON sidecars) — each audio file has a matching JSON file holding the audio filename, metadata, and (optionally) a list of time-aligned utterance records:

Field	Default	Required	Description
`root_strategy`	—	✓	Must be `"paired_glob"`.
`format`	—	✓	Must be `"json"`.
`file_pattern`	—	✓	Glob pattern to find the JSON sidecars (e.g. `"*/.merged.json"`).
`columns`	—	✓	Column mappings over the flattened JSON; nested keys use dot notation (e.g. `audio.filename`, `metadata.speaker2_gender`).
`record_path`	—	✗	Top-level JSON key holding a list of records (e.g. `"transcriptions"`); each record becomes one row and the remaining top-level keys are repeated per row. When omitted, each JSON file yields one row.
`audio_extension`	—	✗	Extension of the paired audio files (e.g. `".wav"`). Pairing normally comes from a filename field inside the JSON, mapped as a `file_path` column with `path_match_strategy: "exact"`.

See the paired-glob strategy for a complete example.

Multi-sections fields

Field	Default	Required	Description
`root_strategy`	—	✓	Must be `"multi_sections"`.
`sections`	—	✓	List of section directory names to load. Unlisted directories are ignored.
`section_root`	—	✓	Directory containing the section subdirectories, relative to the dataset root.
`index_file`	—	✓	Name of the per-section index file, resolved as `section_root/<section>/<index_file>`.

Column mappings are applied to each section's index file when columns is declared (otherwise the raw columns are returned), and a section column with the directory name is added before concatenation.

Glob fields

Field	Default	Required	Description
`root_strategy`	—	✓	Must be `"glob"`.
`file_pattern`	—	✓	Glob pattern to match files (e.g. `"*/.wav"`).
`columns`	—	✗	Mapping of logical column names to path-derived sources: `path`, `name`, `stem`, `parent`, `parents[N]`, `content` (file text). When omitted, the default output is `audio_path`, `language` (parent directory), `speaker_id` (grandparent directory). See the glob strategy page.
`splits`	—	✗	List of subdirectory names to glob through. Each becomes a value in the `split` column. When omitted, the glob runs from the dataset root.

Inner archive extraction

Field	Default	Required	Description
`extract_files`	—	✗	List of archive paths (relative to dataset root) to extract before loading. Supports `.tar.gz`, `.tar.bz2`, `.tar.xz`, and `.zip`. Extraction is skipped on subsequent runs.

This field is task-agnostic — it works with any loader.

Column mapping

Used by every strategy. Each key under columns is the logical column name that will appear in the resulting DataFrame. For the glob strategy, source_column names a path-derived source (path, parent, content, …) instead of an index-file column — see the glob strategy page; for the paired-glob text variant it names one of the derived audio_path / transcription / split sources. For the index-based strategies:

columns:
  audio_path:
    source_column: "path"       # column name in the index file
    dtype: "file_path"          # see dtype table below
  transcription:
    source_column: "sentence"
    dtype: "string"
  speaker_id:
    source_column: "client_id"
    dtype: "category"
    optional: true              # skip silently if the column is missing

For datasets where the index stores an ID instead of the full audio filename, you can opt into search-based file resolution:

base_audio_path:
  - "data/recipes/"
  - "data/giving_gift/"

columns:
  audio_path:
    source_column: "Sentence ID"
    dtype: "file_path"
    path_match_strategy: "exact"   # "direct" (default), "exact", or "contains"
    file_extension: ".wav"         # optional, helps when the index omits the suffix

With path_match_strategy: "exact", the loader searches the configured base_audio_path directories for a matching filename or stem. With "contains", it searches for a filename or relative path containing the source value. If that source value is not unique enough on its own, use path_template to build the real filename from multiple metadata columns:

base_audio_path:
  - "data/recipes/"
  - "data/giving_gift/"

columns:
  audio_path:
    source_column: "Sentence ID"
    dtype: "file_path"
    file_extension: ".wav"
    path_template: "${Speaker ID}_khm_${Sentence ID}.wav"

path_template placeholders reference raw index-file columns exactly as they appear in the metadata, and ${value} refers to the current column's source value.

base_audio_path can use the same placeholder syntax when the containing directory also depends on metadata columns:

base_audio_path: "data/${Split}/"

columns:
  audio_path:
    source_column: "Sentence ID"
    dtype: "file_path"
    file_extension: ".wav"
    path_template: "${Speaker ID}_khm_${value}"

In that example, each row resolves to dataset_root / data/<Split>/<Speaker ID>_khm_<Sentence ID>.wav.

For headerless files (has_header: false), use a positional integer instead of a column name:

columns:
  audio_path:
    source_column: 0
    dtype: "file_path"
  transcription:
    source_column: 1
    dtype: "string"

Supported dtypes

dtype	Behaviour
`string`	Cast to `str` (default).
`file_path`	Resolve to an absolute path. By default this is `dataset_root / base_audio_path / value`, but the loader can also search one or more `base_audio_path` roots when `path_match_strategy` is set, or render filenames and directory roots from metadata columns with `path_template` and templated `base_audio_path` entries.
`file_content`	Like `file_path`, but instead of keeping the resolved path, reads the file and returns its text content. Useful when the index stores paths to transcription files rather than inline text. Supports the same resolution options (`base_audio_path`, `file_extension`, `path_match_strategy`, `path_template`).
`category`	Cast to pandas `Categorical`.
`int`	Numeric coercion → nullable `Int64`.
`float`	Numeric coercion → `float64`.

Column mappings are validated at parse time: an unknown dtype value or an unknown key inside a mapping entry (e.g. dtpye:) raises a ValueError instead of being silently ignored. Unknown top-level schema keys emit a SchemaValidationWarning (with a "did you mean …?" hint for near-misses) and are kept under the extra catch-all, and unknown root_strategy values fail at parse time.

Complete examples

For full examples for each strategy, visit the respective strategy documentation pages under docs/loaders/.

Schema caching

The SDK caches the schema.yaml inside the extracted dataset directory to minimise API calls:

When load_dataset() runs, it obtains the archive checksum from the download plan.
If a local schema.yaml exists and its stored checksum matches the archive checksum, the cached copy is used and no network request needed.
If the checksums differ (dataset was updated) or no cache exists, the schema is fetched from the remote registry and saved locally with the current checksum.
If the remote registry has no schema, a local cache is used as a fallback when available.