Synchronization architecture

This page is the developer reference for how MailDesk moves mail from a provider into Odoo and keeps the open UI fresh. It complements Realtime architecture: this page documents the shipped, stable 4.1.x pipeline (scheduled jobs + bus + the SSOT/cache/queue data model); the realtime page documents the 4.2.0 near-release push/prefetch layer that sits on top of this pipeline.

This page is the developer reference for how MailDesk moves mail from a provider into Odoo and keeps the open UI fresh. It complements Realtime architecture: this page documents the shipped, stable 4.1.x pipeline (scheduled jobs + bus + the SSOT/cache/queue data model); the realtime page documents the 4.2.0 near-release push/prefetch layer that sits on top of this pipeline.

Scope of "stable" here. Everything in §1–§5 is on 18.0 HEAD in maildesk_mail_client (Basic) and maildesk_mail_client_pro (Pro). The 4.2.0 true-push and body-prefetch work is not yet merged — it lives on branches MailDesk-889-realtime-sync-v18 / MailDesk-prefetch-cache-v18, tracked by MRs !237 (Basic) and !124 (Pro). See Realtime architecture for that layer.

1. The shape of the pipeline

Provider (Gmail / Graph / IMAP)
        │  scheduled sync (cron)            ── §3 jobs 1–4
        ▼
maildesk.message_index   ◄── SSOT, identity = unique(account_id, folder, uid)   ── §2.1
        │
        ├─► maildesk.ui_cache         body/metadata TTL cache, advisory-locked     ── §2.2
        │
        └─► maildesk.ingest_queue     durable SSOT → Odoo mail.thread import       ── §2.3
                    │  cron_scan_aliases (index → queue)  +  cron_import (queue → mail.thread)
                    ▼
            Odoo mail.thread / mail.message

Pro adds: maildesk.update_queue  ── outbound mutations (flag/move/delete) back to the provider
Bus: maildesk.bus_orchestrator → bus.bus._sendone(<recordset channel>) → open OWL UI refresh   ── §4

Two flows:

• Inbound (Basic, one-way): provider → maildesk.message_index (SSOT) → maildesk.ui_cache (bodies) and → maildesk.ingest_queue → Odoo mail.thread.
• Outbound (Pro, two-way): a user mutation (read/flag/move/delete) writes maildesk.email_state and enqueues a coalesced job on maildesk.update_queue; per-provider executors apply it back to the provider.

2. Data model — SSOT, cache, queue, lease

2.1 maildesk.message_index — the single source of truth

Every message MailDesk knows about is one row in maildesk.message_index (models/message_index.py). Its identity is the SQL unique constraint:

_sql_constraints = [
    ("account_folder_uid_unique",
     "unique(account_id, folder, uid)",
     "Message UID must be unique per account/folder"),
]

provider is a stored field on the same row (gmail / outlook / imap); the conceptual "delivery key" is (account, provider, folder, uid), but the enforced DB key is (account_id, folder, uid). Writes go through upsert_one(vals), which does an INSERT ... ON CONFLICT (account_id, folder, uid) DO UPDATE so re-seeing a message is idempotent (message_index.py upsert_one). The SSOT stores metadata only (from/to/cc, subject, date, message-id, flags, folder) — never the body.

2.2 maildesk.ui_cache — body/attachment TTL cache

Bodies and attachment lists live in maildesk.ui_cache (models/maildesk_ui_cache.py), one row per indexed message (unique(index_id)), with a cache_until TTL (default +1 h on create; body upserts use maildesk.cache_retention_days, default 30 days). Concurrent writes are serialized with PostgreSQL transaction-level advisory locks:

(pg_advisory_xact_lock(...), auto-released at transaction end.) Daily cleanup (_cron_cleanup_expired_cache) deletes rows past cache_until except entries whose message is linked to an Odoo record (a linked email's body must survive TTL eviction); the SSOT row is never touched, so an evicted body is simply re-fetched from the provider on next open.

2.3 maildesk.ingest_queue — durable SSOT → mail.thread

maildesk.ingest_queue (models/ingest_queue.py) is a durable, retrying queue that materializes selected SSOT rows into Odoo's native mail.thread / mail.message. One row per message (unique(index_id)), ordered priority asc, id asc, with a state machine (pending → in_progress → done | failed | skipped), retry_count, next_try_at, and started_at. Two crons drive it (see §3): cron_scan_aliases promotes indexed messages into the queue; cron_import(batch=40, max_attempts=5) first recovers stuck in_progress jobs (30-min timeout) then processes a batch.

2.4 maildesk.account_lease — multi-worker sync lock

maildesk.account_lease (models/maildesk_account_lease.py) is a DB-backed, time-boxed lease giving one sync per account at a time across cron workers. unique(account_id); try_acquire(account_id, ttl_seconds) does an INSERT ... ON CONFLICT (account_id) DO UPDATE ... WHERE lease_until < now() with lock_timeout = '0ms' (callers never wait — lock contention 55P03/57014 returns False). renew, release, and reap_expired (via FOR UPDATE SKIP LOCKED) round it out. The cron entrypoints pass lease_ttl=300 (5 minutes), so a crashed worker's lease self-expires.

Pro (two-way) additions: maildesk.update_queue coalesces outbound mutations per delivery target, lease-locked, processed every 2 minutes, max 8 attempts with backoff; maildesk.email.link records email↔Odoo-record links and is what protects linked bodies from cache eviction (§2.2). See Architecture and maildesk_mail_client_pro/models/maildesk_update_queue.py.

3. The nine scheduled jobs (Basic, data/ir_cron.xml)

All nine are defined in maildesk_mail_client/data/ir_cron.xml as ir.cron records with state = code. The cadence and the exact server-side call are reproduced verbatim from that file:

Notes worth knowing:

• Jobs 1–4 are per-provider entrypoints on mailbox.account. Gmail uses the History API, Outlook the Graph delta-link, IMAP the UID/CONDSTORE path; each acquires the account lease (lease_ttl=300) before touching the provider. The Outlook sparse-metadata repair (job 4) backfills thin metadata rows the delta path left behind.
• Jobs 5 + 6 are the ingest pipeline (§2.3): job 5 promotes SSOT rows into the queue, job 6 materializes the queue into mail.thread (recovering stuck jobs first).
• Jobs 7 + 8 fill folder history. Job 7 bootstraps a folder's newest messages so it is usable in seconds; job 8's cron body selects up to 5 incremental folders with backfill_completed_at = False, takes a FOR UPDATE SKIP LOCKED lock on them, then calls progressive_backfill() (newest-first, lower priority than new mail) so two workers never backfill the same folder.
• Job 9 is the daily TTL cache cleanup (§2.2).

The in-module note docs/dataflow/sync_flow.md predates this cron set and quotes some older cadences (e.g. "every 5 minutes" / "3 am"). When the two disagree, data/ir_cron.xml is authoritative — the table above is the live schedule.

4. In-app freshness — the bus

Once a sync has indexed a message, the open UI is refreshed in near-real-time over Odoo's websocket bus, not by re-polling. This is UI push, and it only fires after a scheduled sync has done the provider work — it does not itself reach out to the provider.

4.1 Channels are access-gated recordset channels

models/ir_websocket.py overrides ir.websocket._build_bus_channel_list. The frontend may request the string channels "mailbox.account_<id>" and "maildesk.user"; the override maps those to recordset channels the current user is actually allowed to see:

• "mailbox.account_<id>" → the mailbox.account recordset, but only when the user is a mailbox admin or the account is in their access_user_ids (per-mailbox isolation). Otherwise it is dropped.
• "maildesk.user" → the current user's own res.users recordset (private notifications).

Using recordset channels (not raw guessable strings) means the bus subscription is ACL-checked, the way Odoo Mail gates discuss.channel_*.

4.2 Every emission goes through one orchestrator

All bus writes funnel through maildesk.bus_orchestrator (models/orchestration/bus_orchestrator.py), whose dispatch(channel, type, payload) is the single emission boundary. It refuses string channels, refuses any target model other than {"mailbox.account", "res.users"}, and re-checks access (access_user_ids for account events; self-only for user events) before calling bus.bus._sendone(channel, type, payload). The standardized event types it emits:

The legacy BusNotificationAdapter (infrastructure/adapters/bus_notification_adapter.py) is the use-case-side shim that calls these from inside the sync use cases.

5. The three orchestrators

MailDesk's sync control plane is three AbstractModel orchestrators under models/orchestration/, each with a single responsibility:

A weak spot to know: sync_orchestrator._dispatch_gmail / _dispatch_outlook are stubs (return) — Gmail and Outlook do not route through this orchestrator at all; they are driven directly by their dedicated crons (jobs 1, 3, 4). Treat sync_orchestrator as the IMAP control plane specifically.

6. Bootstrap, backfill, and failure handling

• Bootstrap (job 7, bootstrap_if_pending) fills a folder's newest messages first so the inbox is usable in seconds; it leaves the older range to progressive backfill. Folder sync state is tracked on mailbox.folder (sync_state, backfill_completed_at, backfill_fetched_count); a per-folder advisory transaction lock (namespace 48232, pg_try_advisory_xact_lock) prevents two writers from fighting over the same folder (models/mailbox_folder.py, application/use_cases/sync_imap_folder.py).
• Progressive backfill (job 8) walks history newest-first under a FOR UPDATE SKIP LOCKED selection and a per-run UID/time budget, enqueuing at lower ingest priority so new mail always wins. Tunable via the maildesk.backfill_* config parameters (backfill_batch_size, backfill_job_budget_seconds, backfill_max_total_per_folder, backfill_mode, backfill_bootstrap_max).
• Per-account backoff is honored at the orchestrator gate: sync_all() skips accounts with a future backoff_until, and repeated IMAP sync failures escalate to mailbox.account.needs_admin_attention plus a notify_admin_attention bus event to the account owner.

Config parameters (ir.config_parameter, defaults)

7. What 4.2.0 changes (pointer)

4.2.0 keeps this entire pipeline as the fallback path and adds, on top: true Gmail/Outlook push (webhooks that trigger the existing sync crons), IMAP "autopilot" smart polling, and body prefetch into maildesk.ui_cache. Push shortens latency; it does not replace §3's jobs or §4's bus. Full detail and status (not-yet-merged, MRs !237/!124) is in Realtime architecture.