Data Flow and Processing - seojedaperez/IgnisMap GitHub Wiki
This document details the comprehensive data flow and processing architecture of the IgnisMap emergency response system, covering real-time data ingestion, AI-powered analysis, and automated emergency response workflows.
The IgnisMap system processes multiple data streams including satellite fire detection, weather data, biodiversity information, and infrastructure mapping to provide real-time emergency response capabilities. 1
graph TB
subgraph "External Data Sources"
NASA["NASA FIRMS API<br/>Real-time Fire Data"]
Weather["Open-Meteo API<br/>Weather Forecasts"]
GBIF["GBIF API<br/>Biodiversity Data"]
OSM["OpenStreetMap<br/>Infrastructure Data"]
USGS["USGS MODIS<br/>Vegetation Indices"]
Copernicus["Copernicus Climate<br/>Drought Data"]
end
subgraph "Data Ingestion Layer"
EventHub["Azure Event Hubs<br/>satellite-data"]
Functions["Azure Functions<br/>Data Processing"]
end
subgraph "Processing & Analysis"
AI["Azure Cognitive Services<br/>Image Analysis"]
OpenAI["Azure OpenAI<br/>Risk Assessment"]
WindAnalysis["Wind Analysis Service"]
BiodiversityService["Biodiversity Assessment"]
end
subgraph "Data Storage"
CosmosDB["Cosmos DB<br/>8 Containers"]
Storage["Azure Storage<br/>Blob Storage"]
end
subgraph "Real-time Outputs"
Alerts["Fire Alerts"]
TacticalPlans["Tactical Plans"]
ResourceAllocation["Resource Allocation"]
end
NASA --> EventHub
Weather --> Functions
GBIF --> Functions
OSM --> Functions
USGS --> Functions
Copernicus --> Functions
EventHub --> Functions
Functions --> AI
Functions --> OpenAI
Functions --> WindAnalysis
Functions --> BiodiversityService
Functions --> CosmosDB
Functions --> Storage
CosmosDB --> Alerts
CosmosDB --> TacticalPlans
CosmosDB --> ResourceAllocation
The system integrates with NASA FIRMS API for real-time active fire detection globally. 2
sequenceDiagram
participant FIRMS as "NASA FIRMS API"
participant EventHub as "Azure Event Hubs"
participant Functions as "ProcessSatelliteData Function"
participant AI as "Azure Cognitive Services"
participant Cosmos as "Cosmos DB"
FIRMS->>EventHub: Satellite fire data (every 15 min)
EventHub->>Functions: Trigger processing
Functions->>AI: Analyze fire imagery
AI->>Functions: Return analysis results
Functions->>Cosmos: Store in SatelliteData container
Functions->>Cosmos: Create FireAlert if threshold met
Weather data flows through multiple APIs to provide comprehensive meteorological analysis. 3
flowchart LR
subgraph "Weather APIs"
OpenMeteo["Open-Meteo API<br/>✅ Active"]
NOAA["NOAA Weather API<br/>Advanced Data"]
ECMWF["ECMWF Models<br/>Professional"]
Windy["Windy API<br/>Wind Analysis"]
end
subgraph "Processing Services"
WeatherService["Weather Service"]
WindAnalysis["Wind Analysis Service"]
PlanetaryComputer["Planetary Computer<br/>ERA5 + GFS Data"]
end
subgraph "Storage"
WeatherContainer["WeatherData Container<br/>TTL: 7 days"]
end
OpenMeteo --> WeatherService
NOAA --> WeatherService
ECMWF --> WindAnalysis
Windy --> WindAnalysis
PlanetaryComputer --> WindAnalysis
WeatherService --> WeatherContainer
WindAnalysis --> WeatherContainer
The system processes biodiversity data from multiple sources to assess environmental impact. 4
graph TB
subgraph "Biodiversity Sources"
GBIF_API["GBIF API<br/>Species Observations"]
IUCN["IUCN Red List<br/>Conservation Status"]
ESA["ESA WorldCover<br/>Land Cover Data"]
Protected["Protected Areas<br/>Spatial Data"]
end
subgraph "Processing Pipeline"
BiodiversityService["Biodiversity Assessment Service"]
PlanetaryComputer["Planetary Computer Service"]
LandCoverAnalysis["Land Cover Analysis"]
SpeciesRiskAssessment["Species Risk Assessment"]
end
subgraph "Analysis Outputs"
ThreatLevel["Threat Level Assessment"]
EvacuationPriority["Evacuation Priority Zones"]
ConservationAlerts["Conservation Alerts"]
end
GBIF_API --> BiodiversityService
IUCN --> BiodiversityService
ESA --> PlanetaryComputer
Protected --> LandCoverAnalysis
BiodiversityService --> SpeciesRiskAssessment
PlanetaryComputer --> LandCoverAnalysis
SpeciesRiskAssessment --> ThreatLevel
LandCoverAnalysis --> EvacuationPriority
BiodiversityService --> ConservationAlerts
The core processing logic runs on Azure Functions with specific triggers and workflows. 5
flowchart TD
Start["Data Ingestion Trigger"] --> ProcessSatellite["ProcessSatelliteData<br/>Every 15 minutes"]
ProcessSatellite --> AnalyzeRisk["AnalyzeFireRisk<br/>Azure OpenAI Analysis"]
AnalyzeRisk --> GeneratePlan["GenerateTacticalPlan<br/>Organization-specific"]
GeneratePlan --> SendAlerts["SendAlerts<br/>Push & Email"]
WeatherTrigger["Weather Update Trigger"] --> UpdateWeather["UpdateWeatherData<br/>Hourly updates"]
UpdateWeather --> WindAnalysis["Wind Pattern Analysis"]
WindAnalysis --> RiskRecalculation["Recalculate Fire Risk"]
RiskRecalculation --> ResourceAllocation["Update Resource Allocation"]
SendAlerts --> ResourceAllocation
ResourceAllocation --> AuditLog["Log to AuditLog Container"]
The system uses 8 specialized Cosmos DB containers optimized for different data types and access patterns. 6
graph TB
subgraph "High-Throughput Containers"
SatelliteData["SatelliteData<br/>Partition: /satellite<br/>2000 RU/s<br/>TTL: 30 days"]
FireAlerts["FireAlerts<br/>Partition: /organizationId<br/>1000 RU/s"]
WeatherData["WeatherData<br/>Partition: /location<br/>800 RU/s<br/>TTL: 7 days"]
end
subgraph "Standard-Throughput Containers"
Organizations["Organizations<br/>Partition: /organizationType<br/>400 RU/s"]
MonitoringZones["MonitoringZones<br/>Partition: /organizationId<br/>400 RU/s"]
TacticalPlans["TacticalPlans<br/>Partition: /organizationType<br/>400 RU/s"]
ResourceAllocation["ResourceAllocation<br/>Partition: /organizationId<br/>400 RU/s"]
AuditLog["AuditLog<br/>Partition: /organizationId<br/>400 RU/s"]
end
subgraph "Data Flow Patterns"
SatelliteData --> FireAlerts
WeatherData --> FireAlerts
FireAlerts --> TacticalPlans
Organizations --> MonitoringZones
MonitoringZones --> ResourceAllocation
TacticalPlans --> ResourceAllocation
ResourceAllocation --> AuditLog
end
The Azure Integration Service handles CRUD operations across all containers. 7
sequenceDiagram
participant Client as "Client Application"
participant AzureService as "Azure Integration Service"
participant Cosmos as "Cosmos DB"
participant EventHub as "Event Hubs"
participant Functions as "Azure Functions"
Client->>AzureService: Create Fire Alert
AzureService->>Cosmos: Store in FireAlerts container
AzureService->>EventHub: Send fire-alert-created event
AzureService->>Functions: Trigger analysis function
Functions->>Cosmos: Update alert with analysis
AzureService->>Cosmos: Log action in AuditLog
AzureService->>Client: Return created alert
The system uses Azure Event Hubs for real-time data streaming and processing. 8
graph LR
subgraph "Event Sources"
SatelliteEvents["Satellite Data Events"]
WeatherEvents["Weather Update Events"]
AlertEvents["Fire Alert Events"]
UserEvents["User Action Events"]
end
subgraph "Event Hub Processing"
EventHub["emergency-app-events<br/>4 Partitions<br/>7-day retention"]
end
subgraph "Event Consumers"
ProcessingFunctions["Processing Functions"]
AnalyticsFunctions["Analytics Functions"]
NotificationFunctions["Notification Functions"]
AuditFunctions["Audit Functions"]
end
SatelliteEvents --> EventHub
WeatherEvents --> EventHub
AlertEvents --> EventHub
UserEvents --> EventHub
EventHub --> ProcessingFunctions
EventHub --> AnalyticsFunctions
EventHub --> NotificationFunctions
EventHub --> AuditFunctions
The system integrates multiple AI services for comprehensive fire risk analysis.
flowchart TB
subgraph "Input Data"
SatelliteImages["Satellite Imagery"]
WeatherData["Weather Conditions"]
TerrainData["Terrain Information"]
VegetationData["Vegetation Indices"]
end
subgraph "AI Processing Services"
ComputerVision["Azure Computer Vision<br/>Image Analysis"]
TextAnalytics["Text Analytics<br/>Alert Processing"]
AnomalyDetector["Anomaly Detector<br/>Pattern Recognition"]
OpenAI["Azure OpenAI<br/>Risk Assessment"]
end
subgraph "Analysis Outputs"
FireProbability["Fire Probability Score"]
SpreadPrediction["Spread Prediction Model"]
ThreatAssessment["Threat Level Assessment"]
ResponseRecommendations["Response Recommendations"]
end
SatelliteImages --> ComputerVision
WeatherData --> TextAnalytics
TerrainData --> AnomalyDetector
VegetationData --> OpenAI
ComputerVision --> FireProbability
TextAnalytics --> SpreadPrediction
AnomalyDetector --> ThreatAssessment
OpenAI --> ResponseRecommendations
The system implements a hybrid approach with both real and simulated data sources. 9
pie title "Data Source Implementation Status"
"Real Data Sources" : 40
"Simulated Data Sources" : 35
"Hybrid Implementation" : 25
gantt
title Data Source Migration Timeline
dateFormat YYYY-MM-DD
section Phase 1: Critical Data
NASA FIRMS Integration :done, nasa, 2025-01-01, 7d
GBIF Species Data :done, gbif, 2025-01-08, 7d
OpenStreetMap Infrastructure :active, osm, 2025-01-15, 7d
USGS Vegetation Indices :usgs, 2025-01-22, 7d
section Phase 2: Advanced Data
IUCN Red List :iucn, 2025-02-01, 7d
ECMWF Weather Models :ecmwf, 2025-02-08, 7d
Copernicus Drought Data :copernicus, 2025-02-15, 7d
HERE Maps Routing :here, 2025-02-22, 7d
section Phase 3: Optimization
Windy API Wind Analysis :windy, 2025-03-01, 7d
eBird Real-time Fauna :ebird, 2025-03-08, 7d
EFFIS European Fires :effis, 2025-03-15, 7d
Google Places Critical Infrastructure :google, 2025-03-22, 7d
The system monitors critical metrics for data processing performance. 10
graph TB
subgraph "Performance Metrics"
ResponseTime["Alert Response Time<br/>Target: < 30 seconds"]
Accuracy["Fire Detection Accuracy<br/>Target: > 95%"]
Availability["System Availability<br/>Target: 99.9%"]
PlanGeneration["Tactical Plan Generation<br/>Target: < 2 minutes"]
DataLatency["Satellite Data Latency<br/>Target: < 5 minutes"]
end
subgraph "Monitoring Tools"
AppInsights["Application Insights"]
CosmosMetrics["Cosmos DB Metrics"]
FunctionMetrics["Function App Metrics"]
EventHubMetrics["Event Hub Metrics"]
end
ResponseTime --> AppInsights
Accuracy --> FunctionMetrics
Availability --> AppInsights
PlanGeneration --> FunctionMetrics
DataLatency --> EventHubMetrics