Azure Service Fabric - amitbhilagude/userfullinks GitHub Wiki

  1. Options for Serverless computing

    1. AKS
      1. Support container
      2. Supports open-source framework
    2. Azure Service Fabric
      1. Support containers
      2. Good for Microservices or hosting .net application
    3. Azure Functions
      1. Good for event driven
      2. Options for scheduling
    4. Easy to complex-> Azure Functions -> Azure SF -> AKS

  2. Choose Between AKS vs SF

    1. AKS will support Containers only. SF will support containers, Native applications, and guest executables.
    2. AKS will support all languages. SF will support all languages but .Net and Java-based applications with deep integrations. E.g. programmatically control scaling, configurations, states, etc.
    3. SF Support of built-in programming model e.g Stateful Actor
    4. SF Support of Stateful application with no external dependency.
    5. SF supports docker only. AKS supports multiple.

  3. SF is very powerful and almost Microsoft infra like Azure, DevOps, and O365 are hosted on SF but why is it not popular?

    1. Perception is that it doesn't support the other cloud than Azure.
    2. Perception is that vendor lock-in. But SF is open source now like AKS.
    3. Perception is that it only works on the Windows platform. But it supports Linux as well.
    4. Perception is that it is built for .Net. Yes, Net has deep integration but it supports other languages like Java, Python, etc.
    5. Perception is that it required Visual Studio IDE. But it supports other IDEs as well.

  4. Installations

    1. Cloud
    2. On-Prem
    3. Dev workstations
      1. VS 2019 /17 service fabric Tooling feature. This will add the Service Fabric Project type for creating a new project
      2. Service fabric SDK ( Through Web Platform Installer)

  5. Reliable Services

    1. Reliable Services are stateful or Stateless. Reliable Service is like a console application that can manage state or stateless.
    2. Stateful reliable service is used when you would like to store information in Service Fabric instead of external DB. It will reduce the latency.
    3. Recommended to use state less
      1. Ease for autos calling
      2. Web APIs are built as stateless service
    4. This service will be inherited from Statefulservice and you will have state manager object to interact with reliable collections.
    5. Visual Studio has multiple templates for this type.
      1. They are grouped based on.Net Framework and .Net Core.
      2. They are stateful service or ASP.Net Stateful service. ASP.Net stateful service has additional packages related to the asp.net to build web applications.

  6. Reliable connection data structure

    1. Store data into disc
    2. Available data structures 1. Dictionary 2. Queue
    3. Best practices 1. Keep transactions short 2. Dictionary with more keys than single key with big values 3. Timeout exception handling

  7. Actor

    1. Built using Actor pattern
    2. Build on top of Stateful Reliable services.
    3. This pattern is used to manage the multi-threading where each thread will be a single actor. This actor can perform tasks sequentially. If you want to create multi-threading then you can have multiple actors created.
    4. Actor is a stateful reliable service so it provides state manager objects however it has only a reliable dictionary so it is more lightweight compared with reliable services.
    5. Every actor will have a single thread which will have its own storage and reliable collection.
    6. This service will be inherited from Actor Service.
    7. In terms of managing the state(Reliable collection), It supports 3 options. This option is set as an attribute to the class. Three options for state management
      1. Persisted (Highly reliable but low performance). Similar to Strong consistency in Cosmos. it will update all replicas and then respond.
      2. Volatile: It will update active only and respond. Async it will update the replicas
      3. Not Persisted (Highest performance but low reliability). It will not data into the disc but in-memory only.
    8. Optimising performance of state management
      1. State management is a dictionary and you may need to de-serialize the entire dictionary. Deciding the key is important
      2. Good for one actor for one user
    9. No multi-threading is required as we can create actors for each user and requests will send in the queue.

  8. Service Fabric Programming Models. The below types are available in Visual Studio for creating project

    1. Guest Executable Application:

      1. Runs as exe in Service Fabric

    2. Containers

      1. Create a docker image, upload it into Library and run it as container

    3. Reliable Services

      1. Overview

        1. More access to Service Fabric APIs
        2. Advantages: It is reliable means services stay up in an unreliable environment. Availability, Scalability, and Consistency
        3. Only two methods: One for Listening WebAPI port and RunAsync to write business logic.
        4. When to use Reliable services
          1. You want your service's code (and optionally state) to be highly available and reliable.
          2. You need transactional guarantees across multiple units of state (for example, orders and order line items).
          3. Your application’s state can be naturally modeled as Reliable Dictionaries and Queues.
          4. Your application's code or state needs to be highly available with low latency reads and writes.
          5. Your application needs to control the concurrency or granularity of transacted operations across one or more Reliable Collections.
          6. You want to manage the communications or control the partitioning scheme for your service.
          7. Your code needs a free-threaded runtime environment.
          8. Your application needs to dynamically create or destroy Reliable Dictionaries or Queues or whole Services at runtime.
          9. You need to programmatically control Service Fabric-provided backup and restore features for your service’s state.
          10. Your application needs to maintain a change history for its units of the state.
          11. You want to develop or consume third-party-developed, custom state providers.

      2. Stateful

        1. Service manages the state of the Service.

      3. Reliable collections(Only for Stateful services). Reliable collections are set of APIs mentioned in Microsoft.ServiceFabric.Data.Collections managing states. Reliable collection as Dictionary and Queue.

        1. Add Key into Dictionary requires to Create ITrasaction object from StateManager
        2. AddAsync will add into temp dictionary by using ITransaction object created. Keep lock for 4 s.
        3. CommitAsync() will commit record into log. Any change after AddAsync before commit for key will be not be committed as there is already lock.
        4. Read from reliable dictionary is immutable. If we wish to update kay again, it need be created as new object. TO avoid such mistakes use of System.Collections.Immutable class and seal class which is stored in a key.
        5. HasPersistedState is true for Volatile Reliable persistent. That means state will be not stored in disc and improve latency.
        6. Reliable queue can be used if there are multiple producers and consumers.Simmilar to dictionary it also have Enqueueasync and Dequeueasync then commitAsync.
        7. Serialisation and Deseriliastion will be done using IStatemanager serialise object(IStateSerialise). Custom serialisation will be done using by inheriting it.
        8. Reliable collection is specially designed to store data into disc and read it. it is advanced version system.collection and system.concurrentCollections.

      4. Stateless

        1. Similar like any other Application where state managed by external application like DB.

        2. Common example of stateless service, WebAPI called by Front end application and later forward request to the Stateful Service. which response back later.

    4. ASP.Net Core

      1. Service Fabric integrates with ASP.NET Core so you can write both stateless and stateful ASP.NET Core applications that take advantage of Reliable Collections and Service Fabric's advanced orchestration capabilities.
      2. Example Voting Web where Web API are built Stateless API and which will call Stateful service to store voting data into Dictinary.
        1. Create Sealed class by Inheriting from Web in stateless service which has CreateServiceInstanceListeners methods which return Kerstel communication listeners. Uses Microsoft.ServiceFabric.AspNetCore nugetpackage package includes the UseServiceFabricIntegration extension method on IWebHostBuilder that adds Service Fabric–aware middleware. It also have dependency injection statelessservicecontext.
        2. Create Sealed class by Inheriting from Date in statefulservice which has CreateServiceInstanceListeners methods which return Kerstel communication listeners. Uses Microsoft.ServiceFabric.AspNetCore nugetpackage package includes the UseServiceFabricIntegration extension method on IWebHostBuilder that adds Service Fabric–aware middleware. It also have dependency injection of statefulsrvice context and statemanager.
        3. Communcation between two services will be done using Reverseproxy which needs to be enabled during SF deployment and port.
        4. Web Projects will use proxyaddress and port to communicate with backend services. Using httpclient.

    5. Reliable Actors

      1. Build on top of Reliable services. It is independent unit works in single threaded environment. If you have scenarios that some programs needs to run independently of each other. Actor concurrently run as many as tasks. It takes care of becoming active and garbage collection dispose whenever use. It requires to call proxy method to create a actor and call object.
      2. It built using Open sources framework Orleans which uses which uses Virtual Actor Pattern. It is Reliable actor Framework.
      3. It uses Actor Model. In response to a message it receives, an actor can: make local decisions, create more actors, send more messages, and determine how to respond to the next message received. Actors may modify their own private state, but can only affect each other indirectly through messaging
      4. The Reliable Actor framework provides built-in communication for actors and pre-set state persistence and scale-out configurations.
      5. Create class by inheriting Actor Class and expose that class by IActorInterface

  9. Service Fabric Life cycle.

    1. All services within Service Fabric are managed by Service Fabric Runtime. It takes care of creating instances or deleting them.
    2. Every service fabric has Two Main methods and constructors in Program.cs
      1. Constructor to register this instance in Service Fabric Runtime.
      2. Listener method, this will help you to expose this service with port and namespace so that other can communicate.
      3. RunAsync is a method that will have your logic which gets executed when a request is received. This method has a cancellationToken parameter, this parameter will help you to cancel the execution and SF runtime can understand the status.

  10. How storage works for stateful reliable services or Actors.

    1. Additional DB is not required as service fabric stores data in its own disc. It has a state manager instance that exposes Reliable dictionary and Queue to interact with this Data.
    2. Service Fabric creates replicas of this Data and stores it in multiple nodes to make it reliable. In that node one Primary store is active and the rest replicas are in passive mode and SF will decide which one will become based on health.

  11. Scenario and its mapping with SF programming model

    1. Building an e-commerce site that has a couple of APIs.
      1. ProductCatalog API: CRUD operation of ProductCatalog.
      2. BucketAPI: Allow users to add products to their shopping carts.
      3. CheckouAPI: Check out the shopping card and proceed with the order.
    2. This requirement has a couple of storage requirements.
      1. Storage for product catalog
      2. Storage for shopping cart per user.
    3. Implementation
      1. Build a reliable stateless service called API Service to expose APIs to the front end. There is no difference between normal API service and this service except this service will be inherited from a stateless service class.
      2. Build a stateful reliable service called Product Catalog. With this stateful service, you will have a reliable dictionary and queue to store product catalogs. This service will have CRUD operation with a reliable dictionary to add, remove products, etc. it will expose the port and listener so that the API service can call it. It will expose the interface inherited from IService.
      3. Build a stateful Actor service called User Service. With this stateful service, you will have a reliable dictionary to store the user's shopping cart. it accepts Actor Id as User Id so that it will create one actor per user and store its own state. One-click uses this implementation per user to store the payment information.
      4. API Service will use the interfaces exposed to these services and use service fabric proxy to communicate. Note that to communicate between these services, SF provides multiple options like WCF, HTTP, and service remoting. Service remoting is the default option where you can only use service namespace e.g. servicefabric:/service name where SF can take care of routing requests.

  12. Service fabric explorer in workstations

    1. Create nodes
    2. Set replicas
    3. Explore partitions
    4. Create required partition key in base 64 format( to be explored)

  13. Test disaster recovery

    1. Restart one of node in Service fabric explorer
    2. Use of service fabric controlled chaos to simulate the failure and test

  14. Partition, Replicas, and Instances

    1. Service Fabric Explorer
      1. Allow you to test this with one node or 5 nodes.
      2. Allow you to see applications, services, partitions, replicas, etc and restart it.
    2. Replicas vs Instances
      1. Replica terminology used stateful services and Instances are used for stateless service. The replica is more toward DB and stateful had DB
      2. Both will set the replica\instances in different nodes for high availability.
      3. An instance of a stateless service is a copy of the service logic that runs on one of the nodes of the cluster. An instance within a partition is uniquely identified by its InstanceId
      4. A replica of a stateful service is a copy of the service logic running on one of the nodes of the cluster. In addition, the replica maintains a copy of the state of that service.
    3. Service Fabric Project
      1. Service Fabric has another project which keeps the configuration for Service Fabric.
      2. It has information about how many replica sets, and partitions for different nodes e.g. Local with 1 or 5 nodes and also the cloud.
      3. It also has information about which service is stateful and which is stateless.

  15. Application deployment steps using Powershell

    1. Create SF package
    2. Connect to SF cluster. Note: It is easy connect to local cluster. however if you need to connect cluster created in Azure then certificate is used. This certificate was created and also used while creating cluster and same certificate is needed on locally to connect it. This is also applicable when you create a connection with Azure DevOps.
    3. Deploy package
    4. Register App
    5. Create an instance

  16. Service Fabric native microservices vs Containerised Deployment

    1. Container required additional storage
    2. Need to manage using docker image

  17. Hyper V containers

    1. This offering from Microsoft is for more security compared with others. Containers share the host OS kernel and if there is any security compromise in one container it may have a high impact on others. Hyper V container has a dedicated OK kernel per container.

  18. App with container Deployment

    1. Steps for Container
      1. Create a package
      2. Upload into the image store
      3. Create\ upgrade instance 4. Note that the service fabric manifest will have a container host tag with an image to restore from the registry. SF will get to know it doesn't have binary.
    2. Initiate Instance 2. Azure File share enable for state management with storage account 1. With Powershell 2. Storing state Resource Governance

  19. Orchestrations: Takes card deploying, upgrading applications\containers in nodes without human interaction. Takes cares of failover mechanims.

  20. Service Fabric Application Life Cycle

    1. Follows same deployment life cycle like earlier application. Design, develop, Testing, Maintainance etc.
    2. Different roles are defined to manage application e.g. Service Developer(service.menifest), Application Developer(application.menifest), Application Adminstrator(Packaging, override changes from applcation tempates), Operator(Deployment)

  21. Service Fabric Managed Cluster

    1. Currently in preview, Help to provision cluster easily. Existing deployment requires to add public IP, ILB, VMSS, Storage etc.

  22. Service Fabric best practices

    1. There is already a production readiness checklist down which is recommended option.
    2. Application Design
      1. Use of API gateway like Azure APIM, Traefik, Application Gateway, etc.
      2. Recommended to use Stateless service and use of external DB.
      3. Application diagnostics will be done using App Insights
    3. Security
      1. Key Vaults is recommended option for secret management like Certificate
      2. ACL
      3. TLS 1.2
    4. Networking pattern e.g. Load balancer
    5. Capacity Planning
      1. Reliability level and Durability level
      2. Bronze, Silver, Gold, and Platinum which help to define Replicaset and instances node, count
      3. Autoscaling, Verticale scaling and Horizontal scaling
    6. Deployment should be Infrastructure as code i.e. ARM template
    7. Fabric Transport communication for Reliable actors
    8. Primary node should be D2V2 or highly SKU with at least 50 GB HDD capacity
    9. Production cluster must be secure
    10. Add resource constraints at the node level so that they don't consume more than 75% of nodes
    11. Load and Scale test to Identity the capacity requirements
    12. Logging and Monitoring for Service Fabric cluster, VMSS, and Application using Azure Monitor
    13. Cluster must have a primary and secondary certificate
    14. Separate cluster for Dev, staging and Prod
    15. Turn off automatic updates on Prod and turn on Dev and Stage and test
    16. CI-CD pipeline for deployment

  23. Cluster Management

    1. Cluster management using vNet and NSG
    2. Cluster Monitor using Azure Monitor
    3. Cluster scale in,scale-out and Load balancing
    4. Cluster upgrade runtime.
    5. Disaster Recovery

  24. Service Fabric deployment using Powershell

    1. log in to Azure
    2. Create a Resource Group
    3. Create Key Vault
    4. Create a self-signed certificate and upload the certificate in the key vault
    5. Virtual Machine Scale Set
    6. Load balancer
    7. Storage account
    8. Create Service Fabric cluster: Update Certificate thumbprint, Management Endpoint, Cluster Reliability and Durability level

  25. Service Fabric Security

    1. Cluster and server certificate (required)
    2. This is required to communicate nodes securely.
    3. It uses Primary and Secondary(Optional) certificates.
    4. Authentication with management endpoint or Service fabric explorer same certificate will be used.
    5. Subject field used in the certificate cluster must be common name or the Certificate's Subject Alternative Name(SAN)
    6. The certificate must contain a private key. These certificates typically have extensions .pfx or .pem
    7. The certificate must be created for key exchange, which is exportable to a Personal Information Exchange (.pfx) file.
    8. Certificate's subject name must match the domain that you use to access the Service Fabric cluster.
    9. Application certificates (optional)
      1. Encryption and decryption of application configuration values
      2. Encryption of data across nodes during replication.
    10. Client authentication certificates (optional)
      1. Any number of additional certificates can be specified for admin or user-client operations. The client can use this certificate when mutual authentication is required. Client certificates typically are not issued by a third-party CA. Instead, the Personal store of the current user location typically contains client certificates placed there by a root authority. The certificate should have an Intended Purposes value of Client Authentication.
      2. By default the cluster certificate has admin client privileges. These additional client certificates should not be installed into the cluster, but are specified as being allowed in the cluster configuration. However, the client certificates need to be installed on the client machines to connect to the cluster and perform any operations

  26. Cluster security

    1. Cluster consists of nodes and they are connected to each other so that they can communicate with each other. Security is important here so that any malicious activity like Someone will add a new risk node in the cluster or try to hack communication. To avoid this. Node to Node security is important. When a client tries to access the Cluster e.g., Open service explorer, it also needs to be secured. So it will also require Node-Client-Security. You also need the granular permission of the cluster e.g. Admin may have full privileges but someone only need read access to the cluster. The below types of security need to be considered.
    2. Node-to-Node Security
      1. Use of x.509 Certificate and needs to install it on every node. Use Azure Key Vaults.
      2. This certificate needs to be installed on every node and it uses an asymmetric key concept. The public key can be retrieved from Domain Authority.
    3. Node-to-Client Security
      1. Use of X.509 certificate.
    4. Network Security
      1. Create an Isolated environment and use a reverse proxy for public network communication
    5. Application Security
      1. Authentication and authorization.
    6. RBAC
      1. Read only user or Admin user to access cluster.

  27. Recommendation for Cluster Deployment

    1. Not to use the azure portal for Cluster creation but ARM Template(IaaC)
    2. Create a new resource group
    3. Use Azure Key Vault for the certificate
    4. Separate vNet
    5. Use of ARM templates. Use PowerShell if not possible with ARM templates.

  28. Steps

    1. Scripts download https://github.com/aloneguid/pluralsight-using-azureservicefabric-in-production
    2. Install Microsoft Service fabric SDK, PowerShell, Visual Studio code with Azure Resource Manager Tool (display ARM template as Tree) and Powershell extensions.
    3. Run Powershell Scripts from M2 folder. m2.ps1
      1. Checks if an azure account is already logged in. If not, Run Azure-rm-account to login.
      2. Checks if Resource group is already exist, If not, It create new resource group. Update resource group name.
      3. Checks if Key vault is already created, if not new Key vault will be created.
      4. Create self-signed certificate and upload in Key-vaults (Only recommended for Dev and Test). Need to purchase from certificate authority for Prod. It will upload certificate in Key vaults.
      5. Invokes ARM Templates minimal.json to create resources. (Use only for Single node)
        1. Creates Virtual Machine Set
        2. Creates storage accounts for logging and dignosis.
        3. Creates Virtual network for cluster.
        4. Creates load balancer
        5. Creates Public IP address.
        6. Create cluster with single node.
      6. Validate Cluster is created. Manual check and Run script to check.
      7. Two ports are opened in the load balancer. 19000 and 19080. The first port is the management endpoint port exposed for deploying applications in Service Fabric. The second port is the SF explorer port to open Service Fabric explorer.
    4. Plan Cluster Capacity
      1. Minimum 3 nodes in Prod scenario.
      2. Durability vs Reliability
        1. You can set both as Bronze, Silver, and Gold. Platinum is only available for Reliability.
        2. Durability level will give flexibility of infrastructure integration e.g. It will support autoscaling, maintenance updates etc. Reliability is basically how many instances/replicas will be created.
        3. Durability Bronze is a basic setup that will not support autoscaling and maintenance updates. Silver will support autoscaling. Gold will support both and it is the highest level.
        4. Reliability bronze will support 3 nodes, Silver 5 nodes, Gold 7 and platinum supports 9 nodes. VMSS instance count should match to this.
      3. Durability and Reliability types
        1. Need to decide during cluster creation. It has types of Silver, Bronze, Gold, and Platinum. It has predefined nodes count.
        2. Decide based on basic testing and keep 20% more.
        3. Decide node types and Number of Nodes.. Recommended to have multiple node types in Prod. Node types: identical machine with similar CPU, hardware and memory.
    5. Run Powershell and Minimal.json from M3 folder for Silver Durability and Reliability plan. (5 nodes)
    6. Test application 1. Download Test application from https://github.com/aloneguid/pluralsight-using-azureservicefabric-in- production/tree/master/TestApp/Pluralsight.SfProd 2. Run testapp.ps1 to deploy application in cluster.
    7. Create a reverse proxy to enable external access. It is needed because applications deployed in SF, need to open one port per app in ILB. To avoid this you can use a reverse proxy like Traefik or Ngnix which can router requests based on the Url. ILB will have standard 443 port opened. 1. Use Traefik Router to enable. It is another application that needs to host in Service Fabric. Download Traffic router package (officially supported by microsoft. 2. Update the latest package under https://github.com/aloneguid/pluralsight-using-azureservicefabric-in- production/tree/master/Traefik/TraefikPkg 3. Run traefik.ps1 from root path and it will deploy application in service fabric. Application can open using ports mentioned Service fabric. 4. Update the load balancer rule in mimimal.jsonto allow this application. 5. Update servicemenifest.xml in the test application. It allows to creation entry in Traffic application. 6. Run the application to check if it works.

  29. RBAC: This is used to provide read-only and admin access to the service fabric cluster.

    1. Below script with register azure ad apps in AD. you can add users in a group to provide access.
    2. Download the latest AAD tool and copy it https://github.com/aloneguid/pluralsight-using-azureservicefabric-in-production/tree/master/M4
    3. Run prepare app script from M4 to register apps in Azure AD. prepareapps.ps1
    4. Add users in Admin and user roles for the cluster app.
    5. Update the ARM template under M4. refer onenode.json for single node. Refer to Production.json which has everything configured for PROD.

  30. Cluster Monitoring

    1. Cluster monitoring is done using log analytics. All custom logs of Service fabric are stored in Windows events. These event needs to be ingested in log analytics. It is done by installing a Monitoring agent on each node.
    2. create a log analytics
    3. Add Service Fabric log analytics solutions. This plug will help you monitor in a better way.
    4. Once it is created. Go to advanced settings in log analytics and select windows performance counter. i.e. list of metric data can be ingested.

  31. Application Monitoring using Application Insights

  32. Scaling

    1. Manual Scaling of Virtual Machine Scale Sets
    2. Auto scaling of Virtual machine scale sets.