Concepts Background Jobs and Events - osama1998H/Moca GitHub Wiki

Background Jobs & Events

Redis Streams workers, DLQ, scheduler, Kafka/Redis events, transactional outbox, and search sync.

Background Jobs (Redis Streams)

Jobs are processed via Redis Streams with consumer groups:

  • Producer: Any code enqueues a job with queue.Enqueue()
  • Consumer: moca-worker instances read from streams using consumer groups
  • At-least-once delivery: XAutoClaim reclaims unacknowledged messages from crashed workers
  • Dead Letter Queue (DLQ): Failed jobs moved to DLQ after max retries
  • Scheduling: Delayed execution via Redis ZADD (sorted sets)

Queue Priorities

Queue Use Case
default Standard jobs
long Long-running jobs (report generation, bulk operations)
critical High-priority jobs (payment processing, notifications)

Scheduler

moca-scheduler runs as a single-leader process (Redis distributed lock):

  • Reads cron expressions from configuration and app hooks
  • Enqueues scheduled jobs to Redis Streams at trigger time
  • Supports standard cron syntax

Event System

Dual-backend event emitter:

Backend Durability Use Case
Kafka Durable CDC, audit trails, cross-service events
Redis pub/sub Transient Cache invalidation, real-time notifications

Kafka is optional -- Redis pub/sub is the fallback for smaller deployments.

Transactional Outbox

Events are published reliably using the outbox pattern:

  1. Document write + event record inserted in the same DB transaction
  2. moca-outbox polls the outbox table for unpublished events
  3. Publishes to Kafka/Redis, marks as published
  4. Guarantees no events are lost even if the process crashes

Search Sync

Meilisearch indexes are kept in sync via background jobs:

  • Document changes trigger index update jobs
  • Sync daemon processes jobs in bulk
  • Full rebuild available via moca search rebuild

CLI Commands

moca queue status              # Queue health and pending job counts
moca queue list                # List jobs in a queue
moca queue retry               # Retry failed jobs
moca queue purge               # Clear a queue
moca events tail               # Stream events in real-time
moca search rebuild            # Full index rebuild

Related