Schema-Based Dataset Loading

Overview

Every dataset on the Mozilla Data Collective (MDC) platform has an associated schema.yaml file. This declarative file tells the SDK how to turn the raw files inside the archive into a ready-to-use pandas DataFrame, without executing any custom code outside the datacollective library.

from datacollective import load_dataset

df = load_dataset("your-dataset-id")
print(df.head())

Under the hood, load_dataset() performs the following steps automatically:

Resolve the schema: check local cache or the schema registry for schema.yaml. If the dataset is not registered this step raises a warning, so we never download an unsupported archive.
Download the archive (with resume support). The schema we fetched in step 1 tells the loader how the files are structured.
Extract the .tar.gz / .zip to a local directory.
Parse the YAML into a validated DatasetSchema (Pydantic model) and dispatch to the task-specific loader (ASR, TTS, OTH, …), which returns the final DataFrame.

The schema file describes:

What task the dataset is for (ASR, TTS, …).
How to find the data files (index file path, glob pattern, etc.).
How to map raw columns / files into a clean DataFrame.

Minimal example

dataset_id: "common-voice-gsw-24"
task: "ASR"
format: "tsv"
index_file: "train.tsv"
columns:
  audio_path:
    source_column: "path"
    dtype: "file_path"
  transcription:
    source_column: "sentence"
    dtype: "string"

This tells the SDK: "Read train.tsv as tab-separated, take the path column as audio file paths and the sentence column as transcriptions."

Schema fields reference

Required fields

Every schema must have:

Field	Type	Required	Description
`dataset_id`	`str`	✓	Unique dataset identifier on MDC.
`task`	`str`	✓	Task type: determines which loader is used (`"ASR"`, `"TTS"`, `"OTH"`).

Loading strategies

The remaining fields depend on which strategy the dataset uses. The strategy is selected with the root_strategy field; when it is omitted, the loader defaults to the index-based strategy:

Strategy	When to use	Key fields
Index-based (default)	A metadata file (CSV / TSV / pipe-delimited) lists each sample.	`index_file`, `columns`
Multi-split	Multiple split files (train, dev, test, …) each containing samples.	`root_strategy: "multi_split"`, `splits`
Paired-glob	Each audio file has a matching sidecar file (`.txt` for TTS, JSON for ASR), no index file at all.	`root_strategy: "paired_glob"`, `file_pattern`, `audio_extension` (TTS) / `format: "json"`, `record_path`, `columns` (ASR)
Multi-sections	Multiple section directories, each with its own index file.	`root_strategy: "multi_sections"`, `sections`, `section_root`, `index_file`
Glob	Directory-structured dataset with metadata encoded in the path hierarchy.	`root_strategy: "glob"`, `file_pattern`

Index-based fields

Field	Default	Required	Description
`format`	Inferred from `index_file` when possible	✗	Optional format hint: `"csv"`, `"tsv"`, or `"pipe"`. Useful when the file extension is misleading.
`index_file`	—	✓	Path to the metadata file, relative to the dataset root.
`columns`	—	✓	Mapping of logical column names → source columns (see below).
`base_audio_path`	`""`	✗	Directory prefix or list of directories used to resolve `file_path` dtype columns. Entries may also use `${column}` placeholders from the current metadata row.
`separator`	Inferred from `format` or `index_file`	✗	Explicit column separator override (e.g. `"\\|"`).
`has_header`	`true`	✗	Whether the index file has a header row. When `false`, `source_column` must be a positional integer.
`encoding`	`"utf-8"`	✗	File encoding (e.g. `"utf-8-sig"` for files with a BOM).

Multi-split fields

Field	Default	Required	Description
`root_strategy`	—	✓	Must be `"multi_split"`.
`splits`	—	✓	List of split names to load (e.g. `["train", "dev", "test"]`).
`splits_file_pattern`	`"*/.tsv"`	✗	Glob pattern to locate split files.
`columns`	(optional)	✗	Column mappings applied to every split frame.
`base_audio_path`	`""`	✗	Directory prefix or list of directories used to resolve `file_path` dtype columns. Entries may also use `${column}` placeholders from the current metadata row.

Paired-glob fields

TTS (text sidecars) — each audio file has a matching .txt file with the transcription; pairing is done on the filename stem:

Field	Default	Required	Description
`root_strategy`	—	✓	Must be `"paired_glob"`.
`file_pattern`	—	✓	Glob pattern to find text files (e.g. `"*/.txt"`).
`audio_extension`	—	✓	Extension of the matching audio files (e.g. `".webm"`).

ASR (JSON sidecars) — each audio file has a matching JSON file holding the audio filename, metadata, and (optionally) a list of time-aligned utterance records:

Field	Default	Required	Description
`root_strategy`	—	✓	Must be `"paired_glob"`.
`format`	—	✓	Must be `"json"`.
`file_pattern`	—	✓	Glob pattern to find the JSON sidecars (e.g. `"*/.merged.json"`).
`columns`	—	✓	Column mappings over the flattened JSON; nested keys use dot notation (e.g. `audio.filename`, `metadata.speaker2_gender`).
`record_path`	—	✗	Top-level JSON key holding a list of records (e.g. `"transcriptions"`); each record becomes one row and the remaining top-level keys are repeated per row. When omitted, each JSON file yields one row.
`audio_extension`	—	✗	Extension of the paired audio files (e.g. `".wav"`). Pairing normally comes from a filename field inside the JSON, mapped as a `file_path` column with `path_match_strategy: "exact"`.

See ASR loader for a complete example.

Multi-sections fields

Field	Default	Required	Description
`root_strategy`	—	✓	Must be `"multi_sections"`.
`sections`	—	✓	List of section directory names to load. Unlisted directories are ignored.
`section_root`	—	✓	Directory containing the section subdirectories, relative to the dataset root.
`index_file`	—	✓	Name of the per-section index file, resolved as `section_root/<section>/<index_file>`.

Each section's index file is read as-is (no column mappings) and a section column with the directory name is added before concatenation.

Glob fields

Field	Default	Required	Description
`root_strategy`	—	✓	Must be `"glob"`.
`file_pattern`	—	✓	Glob pattern to match files (e.g. `"*/.wav"`).
`splits`	—	✗	List of subdirectory names to glob through. Each becomes a value in the `split` column. When omitted, the glob runs from the dataset root.

Inner archive extraction

Field	Default	Required	Description
`extract_files`	—	✗	List of archive paths (relative to dataset root) to extract before loading. Supports `.tar.gz`, `.tar.bz2`, `.tar.xz`, and `.zip`. Extraction is skipped on subsequent runs.

This field is task-agnostic — it works with any loader.

Column mapping

Used by index-based and multi-split strategies. Each key under columns is the logical column name that will appear in the resulting DataFrame:

columns:
  audio_path:
    source_column: "path"       # column name in the index file
    dtype: "file_path"          # see dtype table below
  transcription:
    source_column: "sentence"
    dtype: "string"
  speaker_id:
    source_column: "client_id"
    dtype: "category"
    optional: true              # skip silently if the column is missing

For datasets where the index stores an ID instead of the full audio filename, you can opt into search-based file resolution:

base_audio_path:
  - "data/recipes/"
  - "data/giving_gift/"

columns:
  audio_path:
    source_column: "Sentence ID"
    dtype: "file_path"
    path_match_strategy: "exact"   # "direct" (default), "exact", or "contains"
    file_extension: ".wav"         # optional, helps when the index omits the suffix

With path_match_strategy: "exact", the loader searches the configured base_audio_path directories for a matching filename or stem. With "contains", it searches for a filename or relative path containing the source value. If that source value is not unique enough on its own, use path_template to build the real filename from multiple metadata columns:

base_audio_path:
  - "data/recipes/"
  - "data/giving_gift/"

columns:
  audio_path:
    source_column: "Sentence ID"
    dtype: "file_path"
    file_extension: ".wav"
    path_template: "${Speaker ID}_khm_${Sentence ID}.wav"

path_template placeholders reference raw index-file columns exactly as they appear in the metadata, and ${value} refers to the current column's source value.

base_audio_path can use the same placeholder syntax when the containing directory also depends on metadata columns:

base_audio_path: "data/${Split}/"

columns:
  audio_path:
    source_column: "Sentence ID"
    dtype: "file_path"
    file_extension: ".wav"
    path_template: "${Speaker ID}_khm_${value}"

In that example, each row resolves to dataset_root / data/<Split>/<Speaker ID>_khm_<Sentence ID>.wav.

For headerless files (has_header: false), use a positional integer instead of a column name:

columns:
  audio_path:
    source_column: 0
    dtype: "file_path"
  transcription:
    source_column: 1
    dtype: "string"

Supported dtypes

dtype	Behaviour
`string`	Cast to `str` (default).
`file_path`	Resolve to an absolute path. By default this is `dataset_root / base_audio_path / value`, but the loader can also search one or more `base_audio_path` roots when `path_match_strategy` is set, or render filenames and directory roots from metadata columns with `path_template` and templated `base_audio_path` entries.
`file_content`	Like `file_path`, but instead of keeping the resolved path, reads the file and returns its text content. Useful when the index stores paths to transcription files rather than inline text. Supports the same resolution options (`base_audio_path`, `file_extension`, `path_match_strategy`, `path_template`).
`category`	Cast to pandas `Categorical`.
`int`	Numeric coercion → nullable `Int64`.
`float`	Numeric coercion → `float64`.

Content mapping

Reserved for future use — content_mapping is accepted by the schema parser but not consumed by any current loader. It is intended for glob-based tasks (e.g. LM) to describe how file contents become DataFrame columns:

content_mapping:
  text: "file_content"     # each file's text → "text" column
  meta_source: "file_name" # filename → "meta_source" column

Complete examples

For full examples for each task and strategy, visit the respective task documentation pages under docs/loaders/.

Schema caching

The SDK caches the schema.yaml inside the extracted dataset directory to minimise API calls:

When load_dataset() runs, it obtains the archive checksum from the download plan.
If a local schema.yaml exists and its stored checksum matches the archive checksum, the cached copy is used and no network request needed.
If the checksums differ (dataset was updated) or no cache exists, the schema is fetched from the remote registry and saved locally with the current checksum.
If the remote registry has no schema, a local cache is used as a fallback when available.