Offline Synchronization Engine

The Offline Synchronization Engine is the backbone of the MQUAD mobile field app, enabling enumerators to collect large volumes of complex survey data and media in completely disconnected environments, with reliable, verified sync when connectivity becomes available.

1. Sync Architecture Overview

The synchronization system is organized into a modular pipeline separating queue state management, network transport, conflict reconciliation, and background isolation.

Diagram source

graph TD
    subgraph Local Device [Local SQLite / Drift Layer]
        Drafts[Draft Records] --> Gate[Interview Submission Gate]
        Gate --> Responses[Responses Table]
        Responses --> Queue[Sync Queue Manager]
    end

    subgraph Sync Engine [Sync Orchestration Layer]
        Queue --> Lock[Sync Lock Service]
        Lock --> Iter[Multi-Tenant Iterator]
        Iter --> Push[Payload Serializer & Chunker]
    end

    subgraph Remote Microservices [MQUAD Backend]
        Push --> SyncAPI[sync-service /push]
        Push --> MediaAPI[media-service /upload]
        SyncAPI --> Rec[Reconciliation Service]
    end

    Rec --> Status[Update Sync Status & Paradata]

Core Responsibilities:

1. Zero-Data-Loss Queuing: Transactional write-ahead transitions ensuring records cannot be dropped even if the app crashes during transmission.
2. Multi-Tenant Iteration: Sequentially or concurrently synchronizing records for multiple assigned organizations using multi-tenant tokens and distinct X-Tenant-ID headers.
3. Resumable Media Uploads: Chunking large audio, image, and video attachments with SHA-256 validation.
4. Bidirectional Reconciliation: Merging server updates, schema refreshes, and rejection corrections without clobbering field-collected responses.

2. Sync Pipeline & Record States

Survey responses move through a strictly monitored state lifecycle:

enum SyncStatus {
  draft,
  completed,
  queued,
  syncing,
  synced,
  conflict,
  error,
}

3. Multi-Tenant Iterative Sync

MQUAD authenticates field workers using a multi-tenant JWT token where tenant_id: null in the base payload and assigned organizations are enumerated in tenant_ids:

// Multi-Tenant Sync Orchestration
Future<SyncResult> syncAllTenants(List<Tenant> assignedTenants) async {
  final results = <String, TenantSyncReport>{};

  for (final tenant in assignedTenants) {
    try {
      // 1. Acquire per-tenant sync lock
      await _syncLockService.acquire(tenant.id);

      // 2. Query pending records for this specific tenant
      final pendingResponses = await _database.getPendingResponses(tenant.id);
      
      // 3. Push records with X-Tenant-ID header
      final report = await _syncService.pushBatch(
        tenantId: tenant.id,
        responses: pendingResponses,
      );
      
      results[tenant.id] = report;
    } finally {
      await _syncLockService.release(tenant.id);
    }
  }
  
  return SyncResult(reports: results);
}

• Each API request includes the headers:
  • Authorization: Bearer <multi_tenant_token>
  • X-Tenant-ID: <target_tenant_uuid>
• The microservices (sync-service, data-service, media-service) use the X-Tenant-ID header to route queries to the correct tenant database.

4. Chunked & Resumable Media Upload

High-resolution photos, audio recordings, and video responses are handled separately from structured JSON payloads to prevent timeouts on low-bandwidth field connections:

1. Local Media Indexing: When an interview captures an audio or photo item, the file is saved to an encrypted sandboxed directory and its path, size, and SHA-256 hash are recorded.
2. Chunking Protocol: Files larger than 1MB are automatically split into 512KB binary chunks.
3. Resumable Upload via media-service:
   • POST /media/upload/init: Pre-registers the file, receiving an upload token.
   • PUT /media/upload/chunk: Sends individual chunks with index and checksum.
   • POST /media/upload/finalize: Server reassembles chunks, verifies SHA-256 integrity, and returns permanent media URI.
4. Retry & Backoff: Interrupted chunk transfers resume from the last acknowledged byte offset rather than restarting from the beginning.

5. Reconciliation & Conflict Resolution Engine

The ReconciliationService manages bidirectional synchronization when remote records are modified or corrected:

Conflict Detection Rules:

• Version Pinning Mismatch: If response.version_id != remote_survey.published_version_id, the engine checks whether schema breaking changes invalidate the record.
• Server-Wins Strategy: If a supervisor in the web portal has approved, archived, or formally rejected a submission, the mobile app accepts the server's authoritative status.
• Client-Wins for In-Flight Edits: If an enumerator has opened an authorized correction session locally, local changes override untouched server drafts.
• Field-Level Diffing: When both local and remote drafts exist, ResponseChangeService compares question keys individually and flags non-overlapping changes for safe auto-merge.

6. Background Scheduling & Isolation

To minimize battery drain while ensuring data synchronizes promptly when entering cellular coverage, MQUAD employs platform-optimized background workers:

Platform Dispatchers:

• Android: Powered by WorkManager using constraints:
  • Network Type: NetworkType.CONNECTED (or UNMETERED when Wi-Fi-only sync is enabled in Settings).
  • Battery: BatteryNotLowConstraint.
• iOS: Uses BGAppRefreshTask registered via Info.plist with minimum background fetch interval of 15 minutes.
• Desktop (macOS / Windows / Linux): Uses a persistent background Dart isolate running on a configurable timer loop.

// Background Task Registration
void registerBackgroundSync() {
  Workmanager().registerPeriodicTask(
    "mquad_periodic_sync",
    "mquad_sync_task",
    frequency: const Duration(minutes: 15),
    constraints: Constraints(
      networkType: NetworkType.connected,
      requiresBatteryNotLow: true,
    ),
  );
}

7. Operational Diagnostics & Anomaly Detection

To protect research veracity, the sync engine continuously evaluates environmental data during synchronization:

• Time Anomaly Detector: Detects device clock tampering (e.g. setting system time back to bypass interview expiration dates) by comparing local system clock against HTTP server Date response headers.
• Compaction Service: Automatically prunes uploaded sync queue logs, clears temporary audio recordings, and reclaims database disk space (VACUUM) every 7 days.
• Paradata Transmission: Collects anonymized telemetry (battery usage, sync duration, network latency, retry counts) and dispatches it alongside survey metadata for remote device health monitoring.