Jobs & Queues

Background job processing for reminders, agendas, attendance, and dental workflows.

Queue Topology

graph LR
    classDef publisher fill:#007AFF0D,stroke:#007AFF,stroke-width:1px;
    classDef redis fill:#00000008,stroke:#0000001A,stroke-width:1px,stroke-dasharray: 4 4;
    classDef worker fill:#007AFF1A,stroke:#007AFF,stroke-width:2px;
    classDef external fill:transparent,stroke:#00000033,stroke-width:1px;

    subgraph Publishers["Event Publishers"]
        direction TB
        API["API Server"]:::publisher
        Cron["Cron Scheduler"]:::publisher
        Web["Webhook Handlers"]:::publisher
    end

    subgraph Redis_["Redis (Railway)"]
        direction TB
        R1[("appointment-reminders")]:::redis
        R2[("daily-agenda")]:::redis
        R3[("dental-messages")]:::redis
    end

    subgraph Workers["BullMQ Workers"]
        direction TB
        W1["Reminder Worker (x5)"]:::worker
        W2["Agenda Worker (x2)"]:::worker
        W3["Dental Worker (x3)"]:::worker
    end

    API --> R1
    Cron --> R2
    Cron --> R3
    Web --> R1
    R1 -->|pop| W1
    R2 -->|pop| W2
    R3 -->|pop| W3
    W1 -.->|dispatch| WhatsApp["Meta WhatsApp API"]:::external
    W2 -.->|dispatch| WhatsApp
    W3 -.->|dispatch| WhatsApp

Cron Schedule

All times in UTC. El Salvador is UTC-6 with no DST.

Job	UTC	CST	Purpose
Daily Agenda	13:00	7 AM	WhatsApp schedule to all active staff
No-Show Mark	05:00	11 PM (prev)	Auto-mark unattended appointments
Attendance Dispatch	/30 * * *	—	QR check-in verification + AI verification
Payment Reminders	14:00	8 AM	Monthly + 3-day advance notices
Recall Nightly	06:00	midnight	Dental recall campaign messages

Idempotency

All appointment-creating jobs use idempotency keys to prevent duplicates:

Source	Key Pattern
Manual booking	`MANUAL_{clinicId}_{doctorId}_{date}_{time}`
WhatsApp booking	`WHATSAPP_{clinicId}_{doctorId}_{date}_{time}_{messageId}`
Reschedule	`RESCHEDULE_{appointmentId}_{newDate}_{newTime}`

Monitoring

Bull Board dashboard at /admin/queues — shows queue depths, job states, failure counts, and retry schedules. Protected by basic auth.

Retry Policies

All three queues share the same defaultJobOptions (backend/src/queues/index.ts) — 3 attempts total, exponential backoff from a 5s base (BullMQ’s exponential strategy is delay * 2^(attemptsMade-1), so the two retry gaps are 5s then 10s before the job is considered exhausted):

Queue	Max attempts	Backoff
appointment-reminders	3	Exponential, 5s base (5s, then 10s between attempts)
daily-agenda	3	Exponential, 5s base (5s, then 10s between attempts)
dental-messages	3	Exponential, 5s base (5s, then 10s between attempts)

Failed jobs are reported to Sentry (Sentry.captureException), not written to audit_logs — see “Alerting & On-Call Runbook” below for exactly what fires and how to find a failed job’s details.

Alerting & On-Call Runbook

What already fires automatically

Every job failure (backend/src/queues/index.ts’s logWorkerFailure) calls Sentry.captureException tagged queue: <queueName>. Once a job exhausts its 3 configured retry attempts (all three queues share the same defaultJobOptions — see “Retry Policies” above), it additionally fires Sentry.captureMessage('[JOB_DEAD_LETTER] ...', 'fatal') — the job then sits permanently visible in Bull Board (removeOnFail: false) for manual inspection.

Alert rule to configure in Sentry

Set an Issue Alert on the backend Sentry project: condition: event message contains [JOB_DEAD_LETTER] and level is fatal → action: notify on-call (Slack/PagerDuty/email, whichever channel is already wired to this Sentry project). This fires only on true exhaustion (all retries used), not on a single transient failure — a job retrying successfully on attempt 2 never reaches this alert.

For earlier warning, a second, lower-urgency rule can watch for queue:appointment-reminders (etc.) tagged exceptions at any rate above baseline — useful for catching a degraded Meta API or Supabase before jobs fully exhaust retries, but not yet configured; add if the dead-letter-only alert proves too late in practice.

Backlog threshold

Bull Board (/admin/queues, basic-auth protected) shows live queue depth per queue; /healthz’s checks.bullmq.queues.{reminder,agenda,dental} also exposes wait/active/ delayed/failed counts per queue programmatically, if a scripted/polling check is ever wired up. No automated backlog-depth alert exists today — for now this is a manual check.

All three queues share the same limiter: { max: 50, duration: 1000 } — the throughput ceiling is 50 jobs/sec per queue, regardless of each worker’s concurrency (5/2/3 — concurrency only bounds how many jobs run simultaneously, not the aggregate rate; BullMQ’s limiter caps completions per window across the whole worker). As a starting threshold: a sustained backlog over ~200 waiting jobs on any single queue for more than 10 minutes implies the worker isn’t keeping up with normal volume (at 50/s that backlog should drain in seconds under healthy conditions) — worth a manual look (Meta API degradation, Redis issue, or a stuck job) even before any job has exhausted retries.

On-call runbook

Alert fires ([JOB_DEAD_LETTER] in Sentry) → open the linked Sentry issue; the queue tag and extra.jobId/extra.data (phone/body/DUI already redacted) identify which job and why it failed (the captured exception message).
Open Bull Board (/admin/queues) → find the failed job by ID in the matching queue → inspect its full failure history (each attempt’s error) and payload.
Check whether the root cause is external (Meta WhatsApp API outage, Supabase outage — check their status pages) or internal (a code bug surfaced by the exception).
If external and now resolved: retry the job manually from Bull Board.
If internal: the job stays in Bull Board (never auto-deleted) as evidence until a fix ships; retry manually once deployed.
If the same failure is recurring across many jobs (not a one-off), check /healthz for the affected dependency (Redis/Supabase) before assuming it’s job-specific.