Realtime & synchronization architecture

This page describes how MailDesk delivers new mail and keeps the UI fresh. It is split deliberately into what ships today (stable) and what is coming in 4.2.0 (near-release), because they are different mechanisms with different guarantees.

This page describes how MailDesk delivers new mail and keeps the UI fresh. It is split deliberately into what ships today (stable) and what is coming in 4.2.0 (near-release), because they are different mechanisms with different guarantees.

Status banner — read this first. The 4.2.0 realtime/prefetch work is not yet merged to 18.0. It lives on branches MailDesk-889-realtime-sync-v18 and MailDesk-prefetch-cache-v18 in both the Basic and Pro repos, tracked by merge requests !237 (Basic) and !124 (Pro). The implementation is complete and test-covered and includes migration 18.0.4.2.0, but 18.0 HEAD does not contain these commits. Treat §2 as forward-looking until the MRs merge; §1 is what runs in shipped 4.1.x builds.

1. Current stable behavior (shipped, 4.1.x)

MailDesk Basic is the synchronization engine; Pro adds outbound (two-way) mutations on top.

1.1 Inbound sync — scheduled, not push

New mail arrives via scheduled background jobs, not provider push. The cron cadence (Basic):

So in stable builds new mail surfaces within the relevant poll interval (≈2–3 min), tunable via the backfill/batch config parameters.

1.2 In-app freshness — the bus

Once a message is indexed, the open UI is updated in near-real-time over Odoo's websocket bus (ir.websocket, account-scoped channels) so other open tabs/users see changes promptly. This is UI push, not provider push — it only fires after a scheduled sync has ingested the message.

1.3 Data model

• maildesk.message_index — single source of truth (SSOT) for message metadata; identity by delivery_key = (account, provider, folder, uid).
• maildesk.ui_cache — TTL cache for bodies/attachment lists, guarded by advisory locks.
• maildesk.ingest_queue — durable SSOT → mail.thread import with retry.
• Pro adds maildesk.update_queue — outbound mutation queue (flag/label/move/soft-delete), coalesced per delivery, lease-locked, processed every 2 min, max 8 attempts with backoff.

Source: Basic models/ir_websocket.py, data/*cron*.xml, models/maildesk_message_index.py, models/maildesk_ui_cache.py; Pro models/maildesk_update_queue.py + per-provider */mutation_executor.py.

2. 4.2.0 near-release behavior

4.2.0 adds three things on top of §1: true push for Gmail and Outlook, IMAP autopilot, and body prefetch. The scheduled jobs in §1 remain as the fallback path.

2.1 What "realtime" means per provider — be precise

Push does not bypass the engine: a webhook stamps the account and triggers the existing sync cron, which then does the delta fetch and bus broadcast. So push shortens latency, it does not replace §1's pipeline.

2.2 IMAP autopilot

A weekly probe (application/use_cases/detect_imap_profile.py) runs CAPABILITY and classifies each IMAP account:

• A — IDLE + CONDSTORE + QRESYNC + UIDPLUS (cheapest deltas)
• B — IDLE only (full flag scan)
• C — basic IMAP (UID append detection)
• D — caps unreliable → strict budget mode

application/use_cases/sync_imap_folder_smart_pro.py then selects the cheapest correct strategy per profile. There are no tuning knobs; classification is deterministic and re-probed weekly (data/ir_cron_imap_profile.xml).

2.3 Body prefetch (all providers)

New model maildesk.message_prefetch_queue warms maildesk.ui_cache so message bodies are ready before the user opens them. Two hooks feed it:

• Post-upsert hook mailbox.account._maildesk_after_message_index_upserted(account_id, folder, uid, index_id, is_new) — called by sync_imap_folder.py / sync_gmail_incremental.py / sync_outlook_delta.py after each upsert; Pro enqueues newly-inserted messages at priority 1 (reason sync_new). It does not itself notify the bus or index — only enqueues.
• Folder-open hook mailbox.account._maildesk_enqueue_folder_open_prefetch(account_id, folder_id, top_index_ids) — when a folder's list renders, the top-N visible messages (default 30, maildesk.prefetch.folder_open_top_n) are enqueued at priority 5 (reason folder_open).

A warming cron (data/ir_cron_prefetch.xml, every 1 min) runs WarmMessageCache with guards: batch 5/tick, 10 s time budget, per-account throttle 200/hour, exponential backoff (10 s → … → 2 h), 8 attempts max, not_found terminal. The base hooks are no-ops in Basic; Pro provides the real implementations.

2.4 Setup, renewal, and failure handling

• Setup wizards: wizards/gmail_push_setup_wizard.py (creates the Pub/Sub topic + users.watch), wizards/outlook_push_setup_wizard.py (creates the Graph subscription). Subscriptions are stored on maildesk.push_subscription.
• Renewal: Gmail watch (7-day max) renewed ~6 days; Outlook /me/messages subscription (~7-day max, registered for 6 days) renewed via PATCH back to ~6 days (application/use_cases/register_outlook_subscription.py SUBSCRIPTION_DURATION = 6 days; application/use_cases/renew_push_subscriptions.py).
• Silence detection: if no push arrives within ~6 h, the account is marked for a full delta resync.
• Graceful degradation: if push fails, MailDesk silently falls back to the scheduled crons of §1; if a prefetch fails, the body is fetched on demand when the user opens the message. No feature-wide kill switch — components handle their own errors.
• Large mailboxes: bootstrap newest-first, incremental capped per run and chunked, backfill progressive and lower-priority, prefetch throttled per account.

Source (branch MailDesk-889-realtime-sync-v18): controllers/gmail_push.py, controllers/outlook_push.py, models/maildesk_message_prefetch_queue.py, application/use_cases/{handle_gmail_push, handle_outlook_push, register_gmail_watch, register_outlook_subscription, renew_push_subscriptions, warm_message_cache, detect_imap_profile, sync_imap_folder_smart_pro}.py, models/maildesk_push_subscription.py, migrations/18.0.4.2.0/post-migration.py.

3. Operational notes

• What an admin configures for 4.2.0: run the Gmail and/or Outlook realtime setup wizard per account; IMAP needs nothing (autopilot is automatic). Prefetch is on after the 4.2.0 migration with sane defaults.
• What a user observes: with push enabled, new Gmail/Outlook mail appears in seconds and opening a folder shows already-loaded bodies for the top messages; IMAP mail appears within the poll interval but opens instantly from cache.
• Verifying: the build_diagnostic_bundle.py use case and the prefetch/push crons under Settings → Technical → Scheduled Actions show queue progress and subscription health.

The development-only flags used to render Pro features on an unlicensed local instance for screenshots are not part of this architecture and are never a customer/admin configuration step. They are documented only in the internal demo/Playwright notes.