Media Export Implementation

The Media Export feature allows downloading all media files associated with a specific survey. Given that surveys can contain thousands of large media assets (images, videos, audio), processing is shifted to a background Celery task.

Key Requirements

1. Background Processing: Media extraction, downloading, and zipping in a Celery worker
2. Chunking Large Files: Zip files exceeding 10 GB are split into multiple parts
3. User Notifications: Email with secure presigned download links on completion
4. Storage Cleanup: Generated zip files deleted from S3 after 5 days
5. Frontend Update: Trigger background task with user notification

Architecture

User clicks Export
       |
       v
  Frontend (DataViewer)
       |
       v
  data-service API -> Celery Worker
       |                    |
       v                    v
  Async task ID       Fetch media from S3
  returned to user         |
                           v
                     Compress & chunk (10GB max)
                           |
                           v
                     Upload zips back to S3
                           |
                           v
                     notification-service -> Email with presigned URLs

Implementation Chunks

Chunk 1: Background Task Skeleton

• Celery worker skeleton in data-service
• Updated data-service/api/core/config.py
• Added boto3 to dependencies
• Configured data-service-worker in dev.compose.yaml

Chunk 2: Media Fetching & Zipping

• Media records fetched for given survey_id
• Assets downloaded and compressed using Python's zipfile module
• 10 GB size check to split into multiple .zip parts
• Parts uploaded to dedicated S3 bucket/folder

Chunk 3: Email Notification & Presigned URLs

• Integration with notification-service
• Secure presigned URLs for uploaded zip parts included in email

Chunk 4: Storage Cleanup

• S3 lifecycle policy or Celery Beat cron job to clean files older than 5 days

Chunk 5: Frontend Integration

• DataViewer component calls async export endpoint
• Toast notification informing user the export is running
• Confirmation that email will be sent on completion

Status

• Celery worker skeleton: Done
• Media zipping with 10GB splitting: Done
• Email notifications: Pending
• Storage cleanup policy: Pending
• Docker verification: Pending