Deployment Instructions - Real-Dev-Squad/feature-flag-backend GitHub Wiki

1. JWT Validation Using RDS Public Key

• All incoming requests are authenticated using JWTs (JSON Web Tokens).
• The JWTs are validated against RDS public key

2. Cross-origin requests are allowed only from trusted domains using this regex:

• ^https?://([a-zA-Z0-9-]+\.)*realdevsquad\.com$

3. Lambda Throttling Based on Request Count

• Lambda functions are automatically throttled when the total number of requests (both successful and failed) exceeds the allowed threshold.
• This prevents abuse and protects downstream systems from overload.

4. Reset limit lambda - We can trigger this manually

• Increase the request limit dynamically.
• Raise Lambda concurrency from zero to allow traffic again when needed.

5. Rate limit lambda - This gets trigger automatically when the pendingLimit in RequestLimit table gets less than 1

• Setting concurrency of all Lambda functions to zero, effectively disabling them.
• This is useful for enforcing throttling or stopping processing temporarily.

6. CORS lambda

• This lambda gets called for all the OPTIONS requests made to the feature flag service
• This will evaluate the origin header and return the dynamic header in the response headers.

7. CloudWatch Logging for All Lambda Functions

• Every Lambda function is configured to push logs to AWS CloudWatch.
• This provides visibility into function executions, errors, and performance.
• 

8. API Gateway Logging Setup

• Logs for API Gateway are not enabled by default and must be manually configured:
• Go to API Gateway → Select Stage → Logs and Tracing
• Enable Error logs by default
• Optionally enable INFO logs for debugging when needed.

9. Authentication for the lambdas

• All Lambda functions require JWT-based authentication, ensuring secure access.

10. Rate limiting logic

• On each API request (OPTIONS / other HTTP methods) we reduce the value of pending limit and once this value becomes less than 1 we make all the lambdas throttle (By setting their concurrency to 0)
• On this the lambdas return 429 as the response code

• Create DDB tables

 ## Tables in DynamoDB
###  featureFlag
  Partition Key - Id (string)
  
###  featureFlagUserMapping
  Partition Key - userId (string)
  Sort Key - flagId (string)
  
###  requestLimit
  Partition Key - limitType (String)
  
  No sort key required here

• Create a pipeline using sam pipeline --bootstrap.

• Add Secret keys in Github Secrets.

  These are defined directly in the workflow or derived dynamically:
  
  
  🔒 Required Secrets (secrets)
  🧑‍💻 Authentication & Authorization
  These are used for assuming IAM roles and configuring AWS credentials:
  
  AWS_ACCESS_KEY_ID:  (Pipeline user access key) -> generated by `sam init pipeline` command
  The access key ID for the IAM user or role used in the pipeline.
  
  AWS_SECRET_ACCESS_KEY:  (Pipeline user secret key)
  The corresponding secret access key.
  
  PIPELINE_EXECUTION_ROLE: (Take from AWS -> Cloudformation select the feature flag backend -> Resources tab you find it there)
  The ARN of the IAM role to assume for packaging and deployment tasks.
  
  CLOUDFORMATION_EXECUTION_ROLE: (same as above in Cloudformation -> resources)
  The IAM role used by CloudFormation to execute the change set during deployment.
  
  📦 Artifact & Stack Configuration
  ARTIFACTS_BUCKET: (same as above in Cloudformation -> resources)
  The S3 bucket where the packaged artifacts (like Lambda code) are stored.
  
  REGION: 
  The AWS region where you’re deploying (e.g., us-east-1).
  
  STACK_NAME: 
  The name of the CloudFormation stack to be created or updated.
  

• Add Permissions listed below for the IAM Role add them manually in (AWS -> IAM Roles -> new role), this is being used by AWS lambdas to access DBs, logs, call other lambdas.

• Store RDS public key in AWS Systems manager - Parameter store (https://github.com/Real-Dev-Squad/feature-flag-backend/issues/143)

• Change the Physical ID of lambda's in template.yaml to the lambda that got created in your Cloudformation stack.

  • These are the ARNs of the lambda functions, these gets used in Rate limit lambda / Reset limit lambda function

  Replace the second value with your Lambda ARN (Select a lambda and you will find the ARN -> Amazon Resource number)

• Enabling Logging in cloudwatch (This has to be done explicitly)

  • - Go to API Gateway → Select Stage → Logs and Tracing

Updated policy as we have also added the auth and reading the public key from the Systems manager parameter store

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "dynamodb:PutItem",
                "lambda:InvokeFunction",
                "dynamodb:GetItem",
                "dynamodb:UpdateItem"
            ],
            "Resource": [
                "arn:aws:dynamodb:us-east-1:502287753636:table/requestLimit",
                "arn:aws:lambda:us-east-1:502287753636:function:feature-flag-RateLimiterFunction-PQZQRZzSKeD0"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:PutLogEvents",
                "ssm:GetParameter"
            ],
            "Resource": "*"
        }
    ]
}

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "VisualEditor0",
			"Effect": "Allow",
			"Action": [
				"dynamodb:PutItem",
				"dynamodb:GetItem",
                                 "dynamodb:UpdateItem",
				"lambda:DeleteFunctionConcurrency",
				"ssm:GetParameter",
				"lambda:PutFunctionConcurrency"
			],
			"Resource": [
				"arn:aws:ssm:us-east-1:502287753636:parameter/*",
				"arn:aws:dynamodb:us-east-1:502287753636:table/requestLimit",
				"arn:aws:lambda:us-east-1:502287753636:function:*"
			]
		}
	]
}

NOTE: We will have to change the AWS account Id in these permissions with the actual AWS account Id in staging / production.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "dynamodb:BatchGetItem",
                "logs:CreateLogStream",
                "dynamodb:BatchWriteItem",
                "dynamodb:PutItem",
                "lambda:InvokeFunction",
                "dynamodb:GetItem",
                "dynamodb:Scan",
                "dynamodb:Query",
                "dynamodb:UpdateItem",
                "ssm:GetParameter",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:ssm:us-east-1:502287753636:parameter/*",
                "arn:aws:logs:us-east-1:502287753636:log-group:*:log-stream:*",
                "arn:aws:dynamodb:us-east-1:502287753636:table/*",
                "arn:aws:lambda:us-east-1:502287753636:function:feature-flag-RateLimiterFunction-PQZQRZzSKeD0"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "logs:CreateLogGroup",
            "Resource": [
                "arn:aws:logs:us-east-1:181388287005:*",
                "arn:aws:logs:us-east-1:502287753636:log-group:*:log-stream:*"
            ]
        }
    ]
}

• We can create a feature flag, see if the entry got created in table

• We can add a user to the feature flag, try fetching this entry

• We can set the pending Limit value to 1 and on next time hitting the API verify if the lambda functions concurrency is set to 0 and they are throttled.

• We can call the resetLimit and verify if the reservedConcurrency is removed and lambdas are using the 20 unreserved concurrency.

• We can verify the logs coming in cloudwatch for both API gateway and lambda functions.