Events Hub - amitbhilagude/userfullinks GitHub Wiki

  1. Create namespace

    1. Pricing Tier
    2. Through Put input
    3. Zone Redundancy

  2. Create Event hub

    1. Partition count
    2. Automatic Capture storage (Optional) supports Blob or Data Lake. Other option to use stream analytics

  3. Create and Receive event

    1. VS 2019, .Net core 3.0, C# 8.0 includes new features of Event hub

  4. Create dedicated cluster with no ingress limit

  5. Stream Analytics for ingesting in Power BI

    1. Create Stream analytics job to ingest data from Event hub to other storage provider like Power BI.

  6. Data move into SQL Data ware house

    1. Using capture blob, Event Hub, Event grid, Azure function, SQL data warehouse https://docs.microsoft.com/en-us/azure/event-hubs/store-captured-data-data-warehouse

  7. Stream data into data brick https://docs.microsoft.com/en-us/azure/azure-databricks/databricks-stream-from-eventhubs?toc=https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fevent-hubs%2FTOC.json&bc=https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fbread%2Ftoc.json

  8. Event Publisher/Producer - Entity sends events to Events Hub

  9. Partition Key - Must match with Publisher so that publisher policy can create different folder structure for each publisher. Partition cannot be changed laters so plan accordingly. We can have max 32 partitions. Can increase using support team.

  10. Partition Key and through put should be 1:1 match.

  11. SAS Token with send permission for Event publisher and SAS Token with read permission for consumer. Second option is RBAC using AD. Best practices if you are using SAS (Scroll down) https://docs.microsoft.com/en-us/azure/event-hubs/authorize-access-shared-access-signature

  12. Event Hub is reactive processing so it will be delivered to consumer when they are available.

  13. Consumer Group: Grouping of all events in single container\view so that application can read them.It will be also used as long term storage. There is default consumer group already created however we can have more. Recommended to have one receiver per consumer group.

  14. Offset: used for reading events from partition from specific point .e.g timestamp. Need to manage programtically

  15. Checkpoint: Track the receive status and make sure only non received are delivered next time.If check point is needed for each and every event then Event hub may impact the performance. Service bus queue is recommended option.

  16. Event processing

    1. Use latest sdk which has event processor .net object (Event Processor Client)
    2. Load balancing and scaling: Create multiple consumer and it also support failover. Event Procesor Client class will take care of it.
    3. Recommended to have multiple consumer group and multiple event processors.
    4. It is thread safe as end to end call from send to processor in synchronous way.

  17. Availability and Consistency

    1. Partition model improves availability. EventHubClient send method takes card of distributing event in partitions
    2. If you need to maintain event order to then recommended to send sequence no or partition key.

  18. Scalability

    1. Depends on 2 factors. Throughput Units and Partitions
    2. Throughput Units: Need to purchase units and each has limit of Ingress and Egress capacity in size or not of events.
    3. Auto inflate feature allow you to increase throuhput unit based on load but no option decrease.

  19. Geo-disaster recovery

    1. Primary and secondary namespace
    2. Enable availability zone

  20. Security

    1. RBAC
    2. SAS: use key vault for keys https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-ip-filtering

  21. Logging Monitoring

    1. Application Insight
    2. Log Analytics
    3. Azure monitor https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-metrics-azure-monitor

  22. Quotas and Limits https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-quotas