Skip to content

Mozilla Data Collective Python SDK Library

Welcome to the documentation for the datacollective Python client for the Mozilla Data Collective REST API.

This library helps you:

  • Authenticate with the Mozilla Data Collective.
  • Download datasets to local storage.
  • Load supported datasets into AI-friendly formats, such as pandas DataFrames and HuggingFace Datasets.

Installation

Install from PyPI with uv:

uv add datacollective

Or with pip:

pip install datacollective

Getting an API Key

To download or upload datasets through the Mozilla Data Collective API, you need an API key:

  1. Sign up to the Mozilla Data Collective platform.
  2. Create or retrieve an API key from your Account -> Credentials page.
  3. Store your API key secret in a .env file, while excluding it from being committed to version control.

Configuration

The client reads configuration from environment variables and .env files.

Environment variables

Required for downloading and uploading datasets:

  • MDC_API_KEY - Your Mozilla Data Collective API key.

Optional:

  • MDC_DOWNLOAD_PATH - Local directory where datasets will be downloaded (defaults to ~/.mozdata/datasets).
  • MDC_LOG_PATH - Directory or file path for the log file (defaults to ~/.mozdata/datacollective.log).

Example using environment variables directly:

export MDC_API_KEY=your-api-key-here
export MDC_DOWNLOAD_PATH=~/.mozdata/datasets
export MDC_LOG_PATH=~/.mozdata

.env file

The client will automatically load configuration from a .env file in the directory you run your code from, or in any of its parent directories.

Create a file named .env:

# MDC API Configuration
MDC_API_KEY=your-api-key-here
MDC_DOWNLOAD_PATH=~/.mozdata/datasets
MDC_LOG_PATH=~/.mozdata

Security note

Do not commit .env files to version control, as they contain secrets.

Logging configuration

The library has the ability to emit detailed logs for debugging and monitoring purposes, on both the console and a local log file. This is disabled by default. To enable logging, pass enable_logging=True to the main functions. When enabled, logs are written to ~/.mozdata/datacollective.log (rotated at 10 MB, 5 backups kept). You can change the log file location by setting the MDC_LOG_PATH environment variable to your desired path, either through the .env file or by executing on your terminal:

export MDC_LOG_PATH=/path/to/save/the/log/files

Basic Usage

IMPORTANT NOTE: Before trying to access any dataset, make sure you have thoroughly read and agreed to the specific dataset's conditions & licensing terms.

Tip

You can find the dataset's ID and/or slug from its page on the MDC platform. Click the Download button and select API / Python Access, then switch to the Python Library tab on the pop-up modal. The Dataset ID and Dataset Slug appear at the bottom of the modal. Both can be used interchangeably in the Python library, but the slug is a more user-friendly identifier.

Download a dataset

Use download_dataset to download a dataset to the configured download path:

from datacollective import download_dataset

archive_path = download_dataset("your-dataset-id")

The function returns the path to the downloaded archive. Pass extract=True to also unpack it into a directory beside the archive, named after the archive without its extension, and get that directory back instead:

dataset_dir = download_dataset("your-dataset-id", extract=True)

An existing archive is reused unless overwrite_existing=True, and an existing extracted directory is reused unless overwrite_extracted=True (or the archive was downloaded again). This is the same directory load_dataset extracts to, so the two never unpack an archive twice.

Note

download_dataset was previously called save_dataset_to_disk. The old name still works for backward compatibility, but it is deprecated and new code should use download_dataset.

The files will be stored under MDC_DOWNLOAD_PATH (default ~/.mozdata/datasets).

Download a dataset's sample

Some publishers provide a sample: a small excerpt of the dataset for evaluation before downloading the full archive. Use download_sample to fetch it:

from datacollective import download_sample

sample_path = download_sample("your-dataset-id")

The sample archive is saved under samples/<dataset-id>/ inside MDC_DOWNLOAD_PATH, apart from the full dataset archives. A sample may share a filename with the full archive, so keeping them in separate directories ensures one is never mistaken for the other.

The function returns the path to the sample archive. Pass extract=True to also unpack it into a directory beside the archive and get that directory back instead:

sample_dir = download_sample("your-dataset-id", extract=True)

If the sample is already present, it is returned without downloading again. Pass overwrite_existing=True to download it again and replace the existing file. With extract=True, an existing extracted directory is likewise reused unless overwrite_extracted=True or the sample was downloaded again.

Note

Downloading a sample requires an API key whose organization has agreed to the dataset's terms on the MDC platform, just like downloading the full dataset. Datasets that publish no sample raise FileNotFoundError. To find datasets with a sample, pass has_sample=True to list_datasets (see Browse and search the dataset catalog).

Programmatic submissions and uploads

The SDK supports creating dataset submissions and uploading files with resumable uploads. The upload state is stored in a JSON file alongside the archive so interrupted uploads can resume automatically.

Note

In order to be able to upload datasets in the MDC platform you will first need to Request Access to Upload by navigating to your profile under the Upload tab. Only the credentials (API keys) created after your request has been approved will be able to upload datasets. Any credentials created before your request was approved will not be able to upload datasets.

from datacollective import (
    DatasetSubmission,
    License,
    Task,
    Visibility,
    create_submission_with_upload,
)

submission = DatasetSubmission(
    name="Dataset Name",
    longDescription="A detailed description of the dataset.",
    shortDescription="A brief description of the dataset.",
    locale="en-US",
    task=Task.ASR,
    format="TSV",
    licenseAbbreviation=License.CC_BY_4_0,
    other="This text should provide a detailed description of the dataset, "
          "including its contents, structure, and any relevant information "
          "that would help users understand what the dataset is about "
          "and how it can be used.",
    restrictions="Any restrictions you want to impose on the dataset",
    forbiddenUsage="Use cases that are not allowed with this dataset",
    additionalConditions="Any additional conditions for using the dataset",
    pointOfContactFullName="Jane Doe",
    pointOfContactEmail="jane@example.com",
    fundedByFullName="Funder Name",
    fundedByEmail="funder@example.com",
    legalContactFullName="Legal Name",
    legalContactEmail="legal@example.com",
    createdByFullName="Creator Name",
    createdByEmail="creator@example.com",
    intendedUsage="Describe the intended usage of the dataset, including "
                  "potential applications and use cases.",
    ethicalReviewProcess="Describe the ethical review process that was "
                         "followed for this dataset, including any approvals "
                         "or considerations related to data collection and usage.",
    showContactInfo=False,  # Whether to publicly display the contact information above
    visibility=Visibility.PUBLIC,  # public | private | restricted
    isPaid=False,  # True = the dataset is compensated and requires `basePriceCents`,
                   # False (default) = the dataset is free to access
    exclusivityOptOut=False,  # True = This dataset is non-exclusive to Mozilla Data Collective, 
                              # False = Dataset is exclusively hosted in Mozilla Data Collective
    agreeToSubmit=True,  # True = You confirm that you have the right to submit this dataset and 
                         # that all information provided in the datasheet is accurate. 
                         # Required to be True to complete the submission process
)

response = create_submission_with_upload(
    file_path="/path/to/dataset.tar.gz",
    submission=submission
)

print(response)

For predefined licenses, pass licenseAbbreviation=License.<VALUE> and leave licenseUrl and license unset. For a custom license, pass a custom string to license and optionally include licenseUrl and licenseAbbreviation.

To publish a compensated dataset instead of a free one, set isPaid=True and a basePriceCents price in USD cents (US Dollars). See Pricing for details.

Tip

To also attach an optional sample of your dataset, pass sample_file_path="/path/to/dataset-sample.tar.gz" to create_submission_with_upload, or upload it separately with upload_sample_file(file_path=..., submission_id=...).

Tip

To upload a new .tar.gz version to an already approved and published dataset, call upload_dataset_file(file_path=..., submission_id=...) directly. Get the submission ID from Profile → Uploads by opening the approved dataset and copying the value after /profile/submissions/ in the URL. This submission ID is different from the dataset ID.

Tip

If a file upload is interrupted, simply rerun the same function above and the upload will resume from where it left off.

Loading and Querying Datasets

Note

In-memory dataset loading is currently supported only for certain datasets.

You can load supported datasets into memory as a pandas DataFrame for analysis:

from datacollective import load_dataset

df = load_dataset("your-dataset-id")

# Inspect the loaded DataFrame
print(df.head())

Once loaded into a DataFrame, you can use standard pandas operations to filter, aggregate, and analyze the data.

For details on how schema.yaml files drive the loading process, see Schema-Based Dataset Loading.

Return as HuggingFace Dataset object

You can also load supported datasets directly as HuggingFace datasets objects. This requires the optional hf dependency:

uv add "datacollective[hf]"  # or pip install "datacollective[hf]"

Then pass return_format="hf" to load_dataset:

from datacollective import load_dataset

ds = load_dataset("your-dataset-id", return_format="hf")

The returned object is a Dataset, or a DatasetDict keyed by split name (e.g. ds["train"]) if the dataset defines multiple splits. If you want an iterable variant to consume lazily in a training loop, call ds.to_iterable_dataset() on the returned object.

If you call load_dataset(..., return_format="hf") without the hf extra dependency installed, a MissingDependencyError is raised with installation instructions.

Lazily decoding audio with HF's Audio()

Audio columns (e.g. in ASR/TTS datasets) are returned as file path strings. To decode the audio lazily on access, cast the column to the Audio feature type. Audio decoding requires extra dependencies, which you can install with pip install "datasets[audio]" or uv add "datasets[audio]":

from datasets import Audio

ds = ds.cast_column("audio", Audio(sampling_rate=16_000))

# Nothing is decoded yet, decoding happens on access, one example at a time:
sample = ds[0]["audio"]
print(sample["sampling_rate"])  # 16000
print(sample["array"])          # numpy array with the decoded waveform

This keeps memory usage low: the dataset only stores the file paths, and each audio file is decoded (and resampled, if you specify a sampling_rate) at the moment you access it.

Get dataset details

You can retrieve info from the datasheet of a dataset without downloading it. This is a public endpoint, so it works without an API key:

from datacollective import get_dataset_details

info = get_dataset_details("your-dataset-id")
print(info)

Browse and search the dataset catalog

You can list or search all published datasets. Results are paginated and each entry is a DatasetDetails, the same model returned by get_dataset_details.

from datacollective import list_datasets

# Free-text search, narrowed down with filters
page = list_datasets(
    "swahili speech",
    task="ASR",  # a Task enum value or string, or a list of them
    license_abbr="CC0-1.0",  # a License enum value or string, or a list of them
    locale=["sw", "sw-KE"],  # any of the given locales
    sort="newest",
    results_per_page=20,
    page_number=1,
)

print(f"{page.total} datasets match, showing {len(page)}")
for dataset in page:
    size_gb = (dataset.sizeBytes or 0) / 1e9
    print(f"{dataset.id}  {dataset.name}  ({dataset.task}, {dataset.locale}, {size_gb:.1f} GB)")

Other filters: file_format (e.g. "WAV"), upload_date ("today", "thisWeek", "thisMonth", "thisYear"), has_sample=True for datasets with a sample file, pricing ("free" or "compensated") and sort_direction ("asc"/"desc"). results_per_page accepts 1 to 100 results per page (the platform defaults to 24).

To walk through every page, increase page_number until you have collected total items:

from datacollective import list_datasets

page_number, collected = 1, []
while True:
    page = list_datasets(task="TTS", results_per_page=100, page_number=page_number)
    collected.extend(page.items)
    if not page.items or len(collected) >= page.total:
        break
    page_number += 1

Discover the available filter values

The task, locale, license and format values that the catalog can currently be filtered by are exposed as well. Every value except a task is drawn from the published catalog, so it only appears while some dataset carries it. Filtering on a value absent from these lists matches nothing rather than failing:

from datacollective import list_dataset_filters

filters = list_dataset_filters()
print(filters.tasks)     # e.g. ['ASR', 'TTS', 'NLP', ...]
print(filters.locales)   # e.g. ['el', 'en', 'sw', ...]
print(filters.licenses)  # e.g. ['CC0-1.0', 'CC-BY-4.0', ...]
print(filters.formats)   # e.g. ['MP3', 'WAV', 'TSV', ...]

Automatic Download Resume

The SDK automatically handles interrupted downloads. If a download is interrupted for any reason (network error, user cancellation, system shutdown, etc.), the SDK will automatically resume from where it left off when you call download_dataset or load_dataset again.

How it works:

  1. When a download starts, the SDK creates a .checksum file alongside the partial download (.part file) to track the download state: the archive's checksum, the presigned download URL and when it expires.
  2. If the download is interrupted, both files are preserved.
  3. On the next download attempt, the SDK detects the partial download and resumes from the last byte received. While the stored download URL is still valid, the resume reuses it and does not open a new download session, so it does not count towards the API's download rate limit. Once the URL has expired, or if the storage rejects it, the SDK requests a new one and still resumes the partial download.
  4. Once the download completes successfully, both files are automatically removed.

Tip

You don't need to do anything special to enable resume functionality, it works automatically. Just call the same function again after an interruption.

Dropped connections are retried in place. If the connection drops while an archive is streaming, the SDK retries the request against the same download URL, continuing from the bytes already on disk. These retries do not start a new download session, so they do not count towards the API's download rate limit. By default up to 3 attempts are made; pass max_download_attempts to download_dataset, download_sample or load_dataset to change this, e.g. for an unreliable connection:

from datacollective import download_dataset

archive_path = download_dataset("<dataset-id>", max_download_attempts=10)

Only connection-level failures are retried. HTTP errors and manual cancellation fail immediately, and a full download can still be resumed later as described above.

Edge cases handled:

  • If the dataset has been updated since the interrupted download, the SDK detects the checksum mismatch and starts a fresh download.
  • If only partial files exist without proper tracking data, the SDK safely starts a fresh download.

Note

Sample downloads (download_sample) do not resume. The API returns no checksum for samples, so an interrupted sample download starts again from the beginning.

Automatically check for extracted archives

The load_dataset function avoids redundant extraction by automatically detecting existing files. It checks if the dataset archive is already downloaded and the folder is extracted under the same name. This behavior applies when overwrite_existing=False and overwrite_extracted=False. The SDK identifies the data if the extracted folder name matches the archive name without the extension.

API Reference

For a detailed API reference, see the API Reference section of the documentation.

Note

This section is intended for maintainers of the datacollective library.

Tests

Run the full test suite:

pytest -v

Internal SDK development workflows, including live E2E configuration and API endpoint overrides, are documented in Internal Development.

Release Workflow

Check out the Release Workflow document for details on how to publish new versions of the library to PyPI using GitHub Actions.