Release 4.2.0 — Realtime sync, IMAP autopilot, body prefetch

This release adds three things on top of the shipped 4.1.x synchronization engine:

Status: near-release, NOT yet merged to 18.0. The 4.2.0 work lives on two feature branches in both the Basic and Pro repositories — MailDesk-889-realtime-sync-v18 and MailDesk-prefetch-cache-v18 — tracked by merge requests !237 (Basic) and !124 (Pro). The implementation is complete and test-covered and includes migration 18.0.4.2.0, but 18.0 HEAD does not yet contain these commits. Treat this page as forward-looking until the MRs land. Re-confirm merge status at publish time. The Basic diff is small (≈ +156 net lines, mostly no-op base hooks); the Pro diff is substantial (≈ +8960 net lines).

This release adds three things on top of the shipped 4.1.x synchronization engine:

1. True provider push for Gmail and Outlook / Microsoft 365.
2. IMAP autopilot — automatic capability profiling and per-account strategy selection.
3. Body prefetch — bodies warmed into the UI cache before the user opens a message.

The scheduled background jobs from 4.1.x stay in place as the fallback path. Push does not replace the engine: a webhook stamps the account and triggers the existing sync cron, which still does the delta fetch and bus broadcast. Push shortens latency; it does not change what gets synced.

1. What "realtime" means, per provider

Be precise about the guarantee. Only Gmail and Outlook genuinely push; IMAP is smart polling plus fetch-on-open prefetch.

Honest framing for customer-facing copy: call it "near-realtime." Gmail and Outlook push within ~15 s; IMAP mail surfaces within the poll interval but opens instantly thanks to prefetch.

2. IMAP autopilot

A weekly probe (application/use_cases/detect_imap_profile.py, cron data/ir_cron_imap_profile.xml, 1 day) runs CAPABILITY against each IMAP account and classifies it:

application/use_cases/sync_imap_folder_smart_pro.py then selects the cheapest correct strategy for the profile. There are no tuning knobs — classification is deterministic and re-probed weekly.

3. Body prefetch (all providers)

A new model maildesk.message_prefetch_queue warms maildesk.ui_cache so message bodies are ready before the user opens them. Two hooks on mailbox.account feed the queue:

• Post-upsert hook — _maildesk_after_message_index_upserted(account_id, folder, uid, index_id, is_new), called by sync_imap_folder.py / sync_gmail_incremental.py / sync_outlook_delta.py after each upsert. Pro enqueues newly-inserted messages at priority 1 (reason sync_new). The hook only enqueues — it does not itself notify the bus or index.
• Folder-open hook — _maildesk_enqueue_folder_open_prefetch(account_id, folder_id, top_index_ids), fired when a folder list renders. The top-N visible messages (default 30, maildesk.prefetch.folder_open_top_n) are enqueued at priority 5 (reason folder_open).

The base hooks are no-ops in Basic; Pro provides the real implementations.

A warming cron (data/ir_cron_prefetch.xml, every 1 min) runs WarmMessageCache with guards:

4. Setup, renewal, and failure handling

• Setup wizards: wizards/gmail_push_setup_wizard.py creates the Pub/Sub topic + users.watch; wizards/outlook_push_setup_wizard.py creates the Graph subscription. Subscriptions are stored on maildesk.push_subscription. IMAP needs nothing — autopilot is automatic.
• Renewal: Gmail watch (7-day max) is renewed ~6 days; Outlook subscription (3-day default) is renewed via PATCH (application/use_cases/renew_push_subscriptions.py). Cron data/ir_cron_push.xml runs hourly.
• Silence detection: if no push arrives within ~6 h, the account is marked for a full delta resync.
• Graceful degradation: if push fails, MailDesk silently falls back to the scheduled crons of 4.1.x; if a prefetch fails, the body is fetched on demand when the user opens the message. There is no feature-wide kill switch — each component handles its own errors.
• Large mailboxes: bootstrap is newest-first, incremental sync is capped per run and chunked, backfill stays progressive and lower-priority, and prefetch is throttled per account.

5. What an admin / user observes after merge

• Admin: run the Gmail and/or Outlook realtime setup wizard per account. IMAP needs no configuration. Prefetch is on after the 18.0.4.2.0 migration with sane defaults. Queue progress and subscription health are visible under Settings → Technical → Scheduled Actions; the build_diagnostic_bundle.py use case captures a support bundle.
• User: with push enabled, new Gmail/Outlook mail appears in seconds and opening a folder shows already-loaded bodies for the top messages; IMAP mail appears within the poll interval but opens instantly from cache.

6. Source

Branch MailDesk-889-realtime-sync-v18 / MailDesk-prefetch-cache-v18: controllers/gmail_push.py, controllers/outlook_push.py, models/maildesk_message_prefetch_queue.py, models/maildesk_push_subscription.py, application/use_cases/{handle_gmail_push, handle_outlook_push, register_gmail_watch, register_outlook_subscription, renew_push_subscriptions, warm_message_cache, detect_imap_profile, sync_imap_folder_smart_pro}.py, wizards/gmail_push_setup_wizard.py, wizards/outlook_push_setup_wizard.py, migrations/18.0.4.2.0/post-migration.py, crons data/ir_cron_prefetch.xml (1 min), data/ir_cron_imap_profile.xml (1 day), data/ir_cron_push.xml (1 h).

See also Realtime & synchronization architecture for the full stable-vs-4.2.0 split.