Programmatic Uploads

This guide explains how to programmatically upload datasets to the Mozilla Data Collective using the datacollective Python SDK.

Overview

The SDK provides a complete workflow for uploading datasets:

1. Create a draft submission - Initialize a new dataset submission
2. Update submission metadata - Add required metadata fields to the submission
3. Upload the dataset file - Upload your archive using resumable multipart uploads
4. Submit for review - Finalize the submission for review

Additionally, the functionality of uploading a dataset file can be used independently to upload a new archive version to an already approved and published dataset submission. Check the Upload a New File Version to an Approved Dataset section for more details.

The SDK also supports resumable uploads, meaning if an upload is interrupted (network error, system shutdown, etc.), you can resume from where it left off.

Prerequisites

Before uploading, ensure you have:

• An API key from the Mozilla Data Collective dashboard
• Your dataset packaged as an archive file (.tar.gz, uploads use application/gzip)
• All the required metadata for the dataset submission

Configuration

Set your API key as an environment variable:

export MDC_API_KEY=your-api-key-here

Or create a .env file in your project directory:

MDC_API_KEY=your-api-key-here

Quick Start: One-Step Upload

The simplest way to upload a dataset is using create_submission_with_upload, which handles the entire workflow in a single call:

from datacollective import (
    DatasetSubmission,
    License,
    Task,
    Visibility,
    create_submission_with_upload,
)

submission = DatasetSubmission(
    name="Dataset Name",
    longDescription="A detailed description of the dataset.",
    shortDescription="A brief description of the dataset.",
    locale="en-US",
    task=Task.ASR,
    format="TSV",
    licenseAbbreviation=License.CC_BY_4_0,
    other="This text should provide a detailed description of the dataset, "
          "including its contents, structure, and any relevant information "
          "that would help users understand what the dataset is about "
          "and how it can be used.",
    restrictions="Any restrictions you want to impose on the dataset",
    forbiddenUsage="Use cases that are not allowed with this dataset",
    additionalConditions="Any additional conditions for using the dataset",
    pointOfContactFullName="Jane Doe",
    pointOfContactEmail="jane@example.com",
    fundedByFullName="Funder Name",
    fundedByEmail="funder@example.com",
    legalContactFullName="Legal Name",
    legalContactEmail="legal@example.com",
    createdByFullName="Creator Name",
    createdByEmail="creator@example.com",
    intendedUsage="Describe the intended usage of the dataset, including "
                  "potential applications and use cases.",
    ethicalReviewProcess="Describe the ethical review process that was "
                         "followed for this dataset, including any approvals "
                         "or considerations related to data collection and usage.",
    showContactInfo=False,  # Whether to publicly display the contact information above
    visibility=Visibility.PUBLIC,  # public | private | restricted
    exclusivityOptOut=False,  # True = This dataset is non-exclusive to Mozilla Data Collective, 
                              # False = Dataset is exclusively hosted in Mozilla Data Collective
    agreeToSubmit=True,  # True = You confirm that you have the right to submit this dataset and 
                         # that all information provided in the datasheet is accurate. 
                         # Required to be True to complete the submission process
)

response = create_submission_with_upload(
    file_path="/path/to/dataset.tar.gz",
    submission=submission
)

print(response)

For predefined licenses, pass licenseAbbreviation=License.<VALUE> and leave licenseUrl and license unset. For a custom license, pass a custom string to license and optionally include licenseUrl and licenseAbbreviation.

Visibility

visibility controls who can access the dataset and must be one of the Visibility enum values:

Value	Visible to:	Who can download:
Visibility.PUBLIC	Everyone	Everyone
Visibility.PRIVATE	Everyone	Your organization & Approved requesters only
Visibility.RESTRICTED	Your organization	Your organization (via SDK)

Upload a New File Version to an Approved Dataset

Use upload_dataset_file when the dataset already exists on the platform and is already in the Published / Approved state.

1. Go to Profile → Uploads on the platform.
2. Click on the dataset submission you want to upload a new version for (must be in an Approved state).
3. Copy the ID from the URL, for example: https://mozilladatacollective.com/profile/submissions/<ID>
4. Pass that value to upload_dataset_file as submission_id.

[!IMPORTANT] The value after /profile/submissions/ is the submission ID, not the dataset ID.

from datacollective import upload_dataset_file

approved_submission_id = "XXXXXXXXXXXXXXXXX"  # submission ID, not dataset ID

upload_state = upload_dataset_file(
    file_path="/path/to/new-dataset-version.tar.gz",
    submission_id=approved_submission_id,
)

print(f"Version upload complete! File Upload ID: {upload_state.fileUploadId}")

If the upload is interrupted, rerun the same call and the SDK will resume from the saved state file.

Required Submission Fields

For a detailed explanation of the required fields in the DatasetSubmission model, see the API Reference section.

To complete the submission process, the submission must include at least all of the following fields:

• name
• longDescription
• task
• locale
• format
• licenseAbbreviation or license
• restrictions
• forbiddenUsage
• pointOfContactFullName
• pointOfContactEmail
• showContactInfo
• visibility
• agreeToSubmit=True

A completed file upload must also be attached to the submission before it can be submitted for review. The uploaded archive is linked to the submission automatically when the multipart upload completes (the upload is started with the submission's ID).

Step-by-Step Upload

For more control over the upload process, you can use the individual functions:

Step 1: Create a Draft Submission

from datacollective import DatasetSubmission, create_submission_draft

submission = DatasetSubmission(
    name="Dataset Name",
    longDescription="A detailed description of the dataset.",
)

draft = create_submission_draft(submission)

submission_id = draft["submission"]["id"]
print(f"Created draft submission: {submission_id}")

Which should output something like:

Created draft submission: cmmjpewijXXXXXXXXX

Step 2: Update Submission Metadata

Fill in the datasheet fields for your submission. Set the metadata before uploading the file, so that if the upload is later interrupted or fails, the submission still has its metadata filled in.

from datacollective import (
    DatasetSubmission,
    License,
    Task,
    Visibility,
    update_submission,
)

update_fields = DatasetSubmission(
    task=Task.ASR,
    licenseAbbreviation=License.CC_BY_4_0,
    locale="en-US",
    format="TSV",
    restrictions="No restrictions.",
    forbiddenUsage="Do not use for unlawful purposes.",
    pointOfContactFullName="Jane Doe",
    pointOfContactEmail="jane@example.com",
    showContactInfo=False,
    visibility=Visibility.PUBLIC,
    # ... other metadata fields ...
)

response = update_submission(
    submission_id=submission_id,
    submission=update_fields,
)

print(f"Metadata updated: {response}")

Step 3: Upload the Dataset File

Then, you can use the submission ID above to upload the dataset file:

from datacollective import upload_dataset_file

upload_state = upload_dataset_file(
    file_path="/path/to/your/dataset.tar.gz",
    submission_id=submission_id,
)

print(f"Upload complete! File Upload ID: {upload_state.fileUploadId}")

[!TIP] You can also find your submission ID by going to your Uploads in your profile, click on the dataset submission of your choice, and the URL will contain the submission ID (e.g., https://mozilladatacollective.com/submissions/cmmjpewijXXXXXXXXX).

Step 4: Submit for Review

from datacollective import DatasetSubmission, submit_submission

response = submit_submission(
    submission_id=submission_id,
    submission=DatasetSubmission(agreeToSubmit=True),
)

submission = response["submission"]
print(f"Submission status: {submission['status']}")

Tuning the Part Size

Uploads are multipart: the file is split into fixed-size chunks ("parts") that are uploaded one by one. Both upload_dataset_file and create_submission_with_upload accept a part_size argument (in bytes) to control this. It defaults to 10 MB.

upload_dataset_file(
    file_path="/path/to/dataset.tar.gz",
    submission_id=submission_id,
    part_size=20 * 1024 * 1024,  # 20 MB parts
)

Why change it?

• Larger parts mean fewer parts overall, so fewer requests and less per-part overhead — useful on fast, reliable connections and for very large files (see the limit below).
• Smaller parts give finer-grained resume: after an interruption only the unfinished part is re-uploaded, not a large chunk. Handy on slow or flaky connections. Note that the storage backend requires every part except the last to be at least 5 MB, so that is the practical lower bound.

The MAX_UPLOAD_PARTS limit

A single upload can have at most 10,000 parts. This means part_size must be large enough that ceil(file_size / part_size) ≤ 10,000. The SDK checks this up front and raises a ValueError telling you the minimum part_size to use, rather than failing partway through the upload. For example, a 1 TB file needs parts of at least ~100 MB.

[!NOTE] When resuming an interrupted upload, part_size is ignored — the SDK reuses the part size recorded in the state file so the already-uploaded parts stay valid.

Resumable Uploads

The SDK automatically handles interrupted uploads using a state file.

How It Works

1. When an upload starts, the SDK creates a state file (.mdc-upload.json) alongside your archive
2. The state file tracks which parts have been successfully uploaded
3. If the upload is interrupted, rerunning the same upload call will resume from where it left off
4. Once the upload completes successfully, the state file is removed automatically

Automatic Resume

Simply rerun the same upload call after an interruption.

Using create_submission_with_upload

from datacollective import create_submission_with_upload

# First attempt (interrupted)
response = create_submission_with_upload(
    file_path="/path/to/dataset.tar.gz",
    submission=submission
)
# Second attempt (resumes automatically)
response = create_submission_with_upload(
    file_path="/path/to/dataset.tar.gz",
    submission=submission
)

Using upload_dataset_file

from datacollective import upload_dataset_file

# First attempt (interrupted)
upload_state = upload_dataset_file(
    file_path="/path/to/your/dataset.tar.gz",
    submission_id=submission_id,
)

# Second attempt (resumes automatically)
upload_state = upload_dataset_file(
    file_path="/path/to/your/dataset.tar.gz",
    submission_id=submission_id,
)

Custom State File Location

You can specify a custom location for the state file:

response = create_submission_with_upload(
    file_path="/path/to/dataset.tar.gz",
    submission=submission,
    state_path="/custom/path/upload-state.json",
)

Disabling Resume

To force a fresh upload (ignoring any existing state), simply delete the state file (.mdc-upload.json) created by the SDK before starting another upload.

Error Handling

The SDK raises specific exceptions for common error cases:

Exception	Cause
FileNotFoundError	The specified file path does not exist
ValidationError	Invalid DatasetSubmission or required string inputs
ValueError	Missing or invalid required parameter
PermissionError	API key is invalid or lacks permissions
RuntimeError	Rate limit exceeded or upload failed

Using the DatasetSubmission Model

All submission inputs use the DatasetSubmission Pydantic model, so validation happens as soon as you construct the model (before any network calls are made).

API Reference

For detailed API documentation, see the API Reference section.

Key Functions

• create_submission_with_upload - One-step submission and upload
• create_submission_draft - Create a draft submission
• update_submission - Update submission metadata
• upload_dataset_file - Upload a file to a submission
• submit_submission - Submit a draft for review