Data Storage and Processing Services - seojedaperez/IgnisMap GitHub Wiki

This document details the Microsoft Azure cloud services that power the IgnisMap emergency response system, including their configuration, interconnections, and automated provisioning for fire monitoring, AI analysis, and real-time emergency response coordination.

Service Architecture Overview

The IgnisMap system leverages eight core Azure services organized into critical and supporting tiers based on operational importance and cost impact. 2

Azure Services Architecture

graph TB
    subgraph "Critical Services ($115-150/month)"
        AzureMaps["emergency-app-maps<br/>Azure Maps S1"]
        CosmosDB["emergency-app-cosmos<br/>Cosmos DB"]
        Functions["emergency-app-functions<br/>Function App"]
        OpenAI["emergency-openai<br/>Cognitive Services"]
    end
    
    subgraph "Supporting Services ($50-120/month)"
        EventHub["emergency-app-events<br/>Event Hubs"]
        CogServices["emergency-app-cognitive<br/>Cognitive Services"]
        KeyVault["emergency-app-vault<br/>Key Vault"]
        AppInsights["emergency-app-insights<br/>Application Insights"]
        Storage["emergencystorage<br/>Storage Account"]
    end
    
    subgraph "External Data Sources"
        NASA["NASA FIRMS API"]
        Weather["Open-Meteo API"]
    end
    
    AzureMaps --> Functions
    CosmosDB --> Functions
    Functions --> OpenAI
    Functions --> CogServices
    EventHub --> Functions
    KeyVault --> Functions
    KeyVault --> CosmosDB
    KeyVault --> AzureMaps
    Functions --> AppInsights
    Functions --> Storage
    
    NASA --> EventHub
    Weather --> Functions
Loading

Infrastructure Provisioning

The Azure infrastructure is provisioned through automated shell scripts that create and configure all necessary resources in a single resource group. 3

Provisioning Workflow

flowchart TD
    Start["azure-setup.sh"] --> Login["az login verification"]
    Login --> ResourceGroup["Create emergency-rg<br/>Resource Group"]
    
    ResourceGroup --> Maps["Create emergency-app-maps<br/>S1 SKU"]
    Maps --> Cosmos["Create emergency-app-cosmos<br/>GlobalDocumentDB"]
    Cosmos --> Containers["Create 8 Cosmos containers<br/>Organizations, FireAlerts, etc."]
    
    Containers --> Storage["Create storage account<br/>Standard_LRS"]
    Storage --> Functions["Create emergency-app-functions<br/>Node.js 18, v4"]
    
    Functions --> Cognitive["Create emergency-app-cognitive<br/>CognitiveServices S0"]
    Cognitive --> EventHubs["Create emergency-app-events<br/>Standard SKU"]
    
    EventHubs --> KeyVault["Create emergency-app-vault<br/>Standard SKU"]
    KeyVault --> Secrets["Store connection strings<br/>and API keys"]
    
    Secrets --> Insights["Create emergency-app-insights<br/>Application Insights"]
    Insights --> Config["Configure Function App<br/>environment variables"]
    
    Config --> Output["Generate azure-config.json<br/>configuration file"]
Loading

Data Storage Architecture

Cosmos DB serves as the primary data store with eight specialized containers optimized for different data types and access patterns. 4

Cosmos DB Container Structure

graph TB
    subgraph "EmergencyDB Database"
        Orgs["Organizations<br/>Partition: /organizationType<br/>400 RU/s"]
        Zones["MonitoringZones<br/>Partition: /organizationId<br/>400 RU/s"]
        Alerts["FireAlerts<br/>Partition: /organizationId<br/>1000 RU/s"]
        Plans["TacticalPlans<br/>Partition: /organizationType<br/>400 RU/s"]
        Satellite["SatelliteData<br/>Partition: /satellite<br/>2000 RU/s<br/>TTL: 30 days"]
        Weather["WeatherData<br/>Partition: /location<br/>800 RU/s<br/>TTL: 7 days"]
        Resources["ResourceAllocation<br/>Partition: /organizationId<br/>400 RU/s"]
        Audit["AuditLog<br/>Partition: /organizationId<br/>400 RU/s"]
    end
    
    subgraph "Access Patterns"
        OrgQuery["Query by organization type"]
        ZoneQuery["Query by organization ID"]
        AlertQuery["Query by organization ID"]
        PlanQuery["Query by organization type"]
        SatQuery["Query by satellite source"]
        WeatherQuery["Query by geographic region"]
        ResourceQuery["Query by organization ID"]
        AuditQuery["Query by organization ID"]
    end
    
    Orgs --> OrgQuery
    Zones --> ZoneQuery
    Alerts --> AlertQuery
    Plans --> PlanQuery
    Satellite --> SatQuery
    Weather --> WeatherQuery
    Resources --> ResourceQuery
    Audit --> AuditQuery
Loading

Container Configuration Details

Each Cosmos DB container is configured with specific throughput allocation and partitioning strategy: 5

  • High-throughput containers: SatelliteData (2000 RU/s), FireAlerts (1000 RU/s), WeatherData (800 RU/s)
  • Standard-throughput containers: All others (400 RU/s)
  • TTL-enabled containers: SatelliteData (30 days), WeatherData (7 days)

Event Processing Architecture

Event Hubs Configuration

The Event Hubs namespace emergency-app-events contains the satellite-data event hub configured for high-throughput satellite data ingestion with 4 partitions for parallel processing and 7-day message retention. 6

Data Processing Flow

sequenceDiagram
    participant NASA as "NASA FIRMS API"
    participant EventHub as "satellite-data Event Hub"
    participant Functions as "Azure Functions"
    participant Cosmos as "Cosmos DB"
    participant AI as "Cognitive Services"
    participant Storage as "Azure Blob Storage"
    
    NASA->>EventHub: Satellite fire data
    EventHub->>Functions: ProcessSatelliteData trigger
    Functions->>Storage: Cache raw data
    Functions->>AI: Analyze fire imagery
    Functions->>Cosmos: Store processed data
    Functions->>Functions: GenerateTacticalPlan
    Functions->>Cosmos: Store tactical plans
    Functions->>AppInsights: Log telemetry
Loading

Service Integration Configuration

Function App Environment Variables

The Function App receives configuration through environment variables managed by the provisioning script: 7

Variable Source Purpose
COSMOS_CONNECTION_STRING Cosmos DB primary connection string Database access
AZURE_MAPS_KEY Azure Maps primary key Geospatial services
COGNITIVE_SERVICES_ENDPOINT Cognitive Services endpoint URL AI service access
COGNITIVE_SERVICES_KEY Cognitive Services primary key AI service authentication
APPINSIGHTS_INSTRUMENTATIONKEY Application Insights key Telemetry collection

Key Vault Secret Management

graph LR
    KeyVault["emergency-app-vault"] --> CosmosSecret["CosmosConnectionString"]
    KeyVault --> MapsSecret["AzureMapsKey"]
    KeyVault --> CognitiveSecret["CognitiveServicesKey"]
    
    CosmosSecret --> CosmosDB["emergency-app-cosmos"]
    MapsSecret --> AzureMaps["emergency-app-maps"]
    CognitiveSecret --> CogServices["emergency-app-cognitive"]
Loading

Three critical secrets are stored in Key Vault for secure access across services. 8

Data Processing Services

Azure Functions Processing Pipeline

flowchart TD
    Timer["Timer Trigger<br/>Every 15 minutes"] --> ProcessSat["ProcessSatelliteData"]
    HTTP["HTTP Trigger"] --> AnalyzeFire["AnalyzeFireRisk"]
    EventTrigger["Event Hub Trigger"] --> UpdateWeather["UpdateWeatherData"]
    
    ProcessSat --> NASA["Query NASA FIRMS"]
    ProcessSat --> CosmosWrite["Store in Cosmos DB"]
    
    AnalyzeFire --> OpenAI["Azure OpenAI Analysis"]
    AnalyzeFire --> GeneratePlan["GenerateTacticalPlan"]
    
    UpdateWeather --> WeatherAPI["Open-Meteo API"]
    UpdateWeather --> CosmosWeather["Store Weather Data"]
    
    GeneratePlan --> SendAlerts["SendAlerts Function"]
    SendAlerts --> Notifications["Push/Email Notifications"]
Loading

The system implements five core Azure Functions for data processing: 9

  • ProcessSatelliteData: Processes NASA FIRMS data every 15 minutes
  • AnalyzeFireRisk: AI analysis with Azure OpenAI
  • GenerateTacticalPlan: Creates organization-specific plans
  • SendAlerts: Push and email notifications
  • UpdateWeatherData: Meteorological data updates

Monitoring and Observability

Application Insights Integration

graph TB
    subgraph "Application Insights Monitoring"
        Performance["Performance Monitoring<br/>Function execution times<br/>Resource utilization"]
        Errors["Error Tracking<br/>Exception logging<br/>Failure analysis"]
        Custom["Custom Telemetry<br/>Fire detection accuracy<br/>Response time metrics"]
        Dependencies["Dependency Tracking<br/>External API monitoring"]
    end
    
    Functions --> Performance
    Functions --> Errors
    Functions --> Custom
    Functions --> Dependencies
    
    Performance --> Alerts["Azure Monitor Alerts"]
    Errors --> Alerts
    Custom --> Dashboard["Custom Dashboards"]
    Dependencies --> APIHealth["API Health Status"]
Loading

Application Insights provides comprehensive monitoring through performance tracking, error analysis, custom telemetry, and dependency monitoring. 10

Key Performance Indicators

Metric Target Purpose
Alert response time < 30 seconds Emergency response effectiveness
Fire detection accuracy > 95% Prediction system reliability
System availability 99.9% Service continuity
Tactical plan generation < 2 minutes Operational readiness
Satellite data latency < 5 minutes Real-time monitoring capability

Cost Structure and Optimization

Monthly Cost Breakdown

The Azure infrastructure operates with an estimated monthly cost of $165-270 USD: 11

pie title Monthly Cost Distribution ($165-270 USD)
    "Azure Maps" : 50
    "Cosmos DB" : 35
    "Cognitive Services" : 40
    "Azure Functions" : 15
    "Event Hubs" : 40
    "OpenAI Services" : 40
    "Storage & Others" : 25
Loading
Service Category Cost Range Services Included
Critical Tier $115-150/month Azure Maps ($50-80), Cosmos DB ($25-40), Functions ($10-20), OpenAI ($30-50)
Supporting Tier $50-120/month Cognitive Services ($30-50), Event Hubs ($30-50), Storage/Vault/Insights ($20-30)

Cost Optimization Strategies

The provisioning scripts implement several cost optimization measures: 12

  • Cosmos DB: Session consistency level and optimized RU/s allocation per container
  • Storage: Standard_LRS redundancy for cost-effective local storage
  • Functions: Consumption plan for serverless cost scaling
  • TTL policies: Automatic data expiration for satellite and weather data

Real Data Integration

External Data Sources Integration

graph TB
    subgraph "Real Data Sources"
        NASA["NASA FIRMS API<br/>Fire detection data<br/>Free"]
        GBIF["GBIF API<br/>Biodiversity data<br/>Free"]
        OSM["OpenStreetMap<br/>Infrastructure data<br/>Free"]
        Weather["Open-Meteo API<br/>Weather forecasts<br/>Free"]
        USGS["USGS Earth Explorer<br/>Vegetation indices<br/>Free"]
    end
    
    subgraph "Azure Processing"
        EventHub["Event Hubs<br/>Data ingestion"]
        Functions["Azure Functions<br/>Data processing"]
        Cosmos["Cosmos DB<br/>Data storage"]
    end
    
    NASA --> EventHub
    GBIF --> Functions
    OSM --> Functions
    Weather --> Functions
    USGS --> Functions
    
    EventHub --> Functions
    Functions --> Cosmos
Loading

The system integrates with multiple real data sources for comprehensive fire monitoring and analysis. 13

Notes

This documentation covers the complete Azure-based data storage and processing infrastructure for the IgnisMap emergency response system. The architecture emphasizes real-time data processing, scalability, and cost optimization while maintaining high availability for critical emergency response operations.

The system leverages Azure's serverless and managed services to minimize operational overhead while providing enterprise-grade reliability and performance for emergency response scenarios.

⚠️ **GitHub.com Fallback** ⚠️