Architecture Data Flow - onouyek/MailApe GitHub Wiki

Data Flow

How data moves through the MailApe system.

User Registration Flow

sequenceDiagram
    participant U as User
    participant W as Web App
    participant DB as PostgreSQL
    participant C as Celery
    participant R as Redis
    participant E as Email Service
    
    U->>W: Submit subscription form
    W->>DB: Create Subscriber (confirmed=False)
    Note over DB: Triggers save() method
    DB->>C: Queue confirmation email task
    C->>R: Store task in Redis
    R->>C: Worker picks up task
    C->>E: Send confirmation email
    C->>DB: Update task status
    
    U->>W: Click confirmation link
    W->>DB: Update Subscriber (confirmed=True)
    W->>U: Show confirmation success

Key Data Points:

  • Subscriber record with email and mailing list relationship
  • Confirmation status tracking
  • Automatic email triggering on model save

Source: models.py#L47-61

Message Distribution Flow

sequenceDiagram
    participant O as Owner
    participant W as Web App
    participant DB as PostgreSQL
    participant C as Celery
    participant R as Redis
    participant E as Email Service
    
    O->>W: Create message
    W->>DB: Save Message
    Note over DB: Triggers build_subscriber_messages task
    DB->>C: Queue build task
    C->>R: Store task
    R->>C: Worker processes build task
    C->>DB: Create SubscriberMessage for each confirmed subscriber
    
    loop For each SubscriberMessage
        DB->>C: Queue send task
        C->>R: Store send task
        R->>C: Worker picks up send task
        C->>E: Send individual email
        C->>DB: Update sent timestamp
    end

Process Breakdown:

  1. Message Creation: Owner creates message content
  2. Subscriber Message Generation: System creates individual delivery records
  3. Email Distribution: Each confirmed subscriber gets individual email
  4. Status Tracking: Delivery status recorded per subscriber

Source: models.py#L72-84 and tasks.py#L22-27

Authentication Data Flow

graph TD
    Login[User Login] --> Session[Django Session]
    Session --> Auth[Authentication Middleware]
    Auth --> View[Protected View]
    View --> Check{Owner Check}
    Check -->|Yes| Allow[Access Granted]
    Check -->|No| Deny[403 Forbidden]
    
    Allow --> Model[Model Access]
    Model --> Filter[Filter by Owner]
    Filter --> Results[User's Data Only]

Security Flow:

  • Session-based authentication
  • Owner-based authorization
  • Data filtering by ownership

Source: views.py#L21-23

API Data Flow

sequenceDiagram
    participant C as API Client
    participant A as API View
    participant S as Serializer
    participant M as Model
    participant DB as Database
    
    C->>A: HTTP Request
    A->>A: Authentication Check
    A->>A: Permission Check
    A->>S: Serialize Request Data
    S->>M: Create/Update Model
    M->>DB: Database Operation
    DB->>M: Return Data
    M->>S: Return Model Instance
    S->>A: Serialize Response
    A->>C: HTTP Response

API Features:

  • Authentication required
  • Rate limiting applied
  • Data serialization/deserialization
  • Permission checks per endpoint

Source: serializers.py and views.py#L7-18

Database Transaction Flow

graph TD
    Request[HTTP Request] --> Transaction[Django Transaction]
    Transaction --> Model1[Model Save]
    Model1 --> Signal[Post-Save Signal]
    Signal --> Task[Queue Celery Task]
    Task --> Commit[Transaction Commit]
    
    Commit --> Background[Background Processing]
    Background --> Worker[Celery Worker]
    Worker --> Email[Send Email]
    Worker --> Update[Update Status]

Transaction Safety:

  • Model saves trigger tasks after transaction commit
  • Failed tasks don't affect web request
  • Eventual consistency for email delivery

Email Processing Pipeline

flowchart TD
    Message[New Message] --> Build[build_subscriber_messages]
    Build --> Query[Query Confirmed Subscribers]
    Query --> Create[Create SubscriberMessage Records]
    
    Create --> Queue[Queue Send Tasks]
    Queue --> Worker1[Celery Worker 1]
    Queue --> Worker2[Celery Worker 2]
    Queue --> WorkerN[Celery Worker N]
    
    Worker1 --> Send1[Send Email]
    Worker2 --> Send2[Send Email]
    WorkerN --> SendN[Send Email]
    
    Send1 --> Status1[Update Sent Status]
    Send2 --> Status2[Update Sent Status]
    SendN --> StatusN[Update Sent Status]

Parallel Processing:

  • Multiple workers process emails concurrently
  • Individual delivery tracking
  • Retry mechanisms for failed sends

Source: tasks.py

Configuration Data Flow

graph TD
    Env[Environment Variables] --> Settings[Django Settings]
    Settings --> Apps[Django Applications]
    
    Static[Static Files] --> Collect[collectstatic]
    Collect --> Serve[Web Server]
    
    DB_Config[Database Config] --> ORM[Django ORM]
    Redis_Config[Redis Config] --> Celery[Celery Workers]
    
    AWS[AWS Resources] --> CloudFormation[CloudFormation]
    CloudFormation --> Infrastructure[Infrastructure]

Configuration Sources:

  • Environment variables for secrets
  • Settings files for application config
  • CloudFormation for infrastructure

Source: common_settings.py#L23

Error Handling Flow

graph TD
    Error[Error Occurs] --> Type{Error Type}
    
    Type -->|Web Error| Web[Web Error Handler]
    Type -->|Task Error| Task[Celery Retry]
    Type -->|DB Error| DB[Transaction Rollback]
    
    Web --> Log[Log Error]
    Task --> Retry[Retry Task]
    DB --> Log
    
    Retry --> Success{Success?}
    Success -->|Yes| Complete[Task Complete]
    Success -->|No| Failed[Mark Failed]
    
    Log --> Monitor[Monitoring System]

Error Recovery:

  • Automatic task retries
  • Transaction rollbacks for data integrity
  • Comprehensive logging

Monitoring Data Flow

graph LR
    App[Application] --> Logs[Log Messages]
    Tasks[Celery Tasks] --> Results[Task Results]
    
    Logs --> Level{Log Level}
    Level --> Console[Console Output]
    Level --> File[Log Files]
    
    Results --> Database[(Database)]
    Database --> Status[Task Status]
    
    Status --> Monitor[Monitoring]
    Console --> Monitor

Observability:

  • Django logging framework
  • Celery result tracking
  • Task status monitoring

Source: common_settings.py#L134-152

Data Persistence Patterns

Write Operations

  • Immediate: User data (registration, list creation)
  • Eventual: Email delivery status
  • Transactional: Related model updates

Read Operations

  • Filtered: Users see only their own data
  • Optimized: Manager methods for efficient queries
  • Cached: Session data in Redis

Data Consistency

  • Strong: Within database transactions
  • Eventual: Across async tasks
  • Compensating: Error recovery mechanisms

Next Steps