production_deployment_checklist - mamoorkhan/glasslewis GitHub Wiki

Production Deployment Checklist

Comprehensive checklist to ensure smooth and secure production deployments of the GlassLewis Platform.

🎯 Overview

This checklist covers:

Pre-deployment preparations and validations
Deployment execution steps
Post-deployment verification and monitoring
Rollback procedures if issues arise

🔍 Pre-Deployment Checklist

📋 Code Quality & Testing

All tests pass locally

dotnet test --configuration Release
cd clients/glasslewis.client.angular && ng test --watch=false

Code coverage meets requirements (>80%)

dotnet test --collect:"XPlat Code Coverage"

Security scan completed

# Example using GitHub Security
git push origin main  # Triggers security scanning

Performance tests pass
```
cd tests/performance && dotnet test
```

E2E tests pass against staging environment

cd tests/ui && dotnet test --settings:staging.runsettings

🏗️ Infrastructure Preparation

Azure resource group exists

az group show --name rg-glasslewis-prod --query name

Bicep templates validated

az deployment group validate \
  --resource-group rg-glasslewis-prod \
  --template-file infra/main.bicep \
  --parameters @infra/params/prod.bicepparam

DNS configuration ready (if using custom domain)
```
nslookup your-custom-domain.com
```

SSL certificates validated

openssl s_client -connect your-domain.com:443 -servername your-domain.com

🔐 Security Configuration

App registrations configured for production URLs
- API app registration: Audience configured
- Client app registration: Redirect URIs updated
- CORS origins include production domains

Secrets and environment variables reviewed

# Verify all required secrets exist in Azure Key Vault
az keyvault secret list --vault-name kv-glasslewis-prod

Service principal permissions validated

az role assignment list --assignee "your-service-principal-id" --scope "/subscriptions/your-sub-id"

Network security groups configured

az network nsg list --resource-group rg-glasslewis-prod

📊 Monitoring & Observability

Application Insights configured

az monitor app-insights component show --app ai-glasslewis-prod --resource-group rg-glasslewis-prod

Health check endpoints functional

curl -k https://api-staging.glasslewis.com/health

Alerting rules configured

az monitor metrics alert list --resource-group rg-glasslewis-prod

Log Analytics workspace ready

az monitor log-analytics workspace show --workspace-name law-glasslewis-prod --resource-group rg-glasslewis-prod

🚀 Deployment Execution

⚡ Automated Deployment (Recommended)

Prerequisites:

GitHub Actions workflow tested in staging
All secrets configured in GitHub repository
Branch protection rules configured

Execution Steps:

Create Release Branch

git checkout main
git pull origin main
git checkout -b release/v1.0.0
git push origin release/v1.0.0

Create Release Tag

git tag -a v1.0.0 -m "Production release v1.0.0"
git push origin v1.0.0

Monitor GitHub Actions
- Go to GitHub → Actions tab
- Watch deployment pipeline progress
- Verify each stage completes successfully
Deployment Pipeline Stages:
- Build: Code compilation and packaging
- Test: Automated test execution
- Security: Security scanning and validation
- Infrastructure: Azure resource deployment
- API Deployment: Backend service deployment
- Frontend Deployment: Angular app deployment
- Smoke Tests: Basic functionality validation

🔧 Manual Deployment (Backup Method)

Only use if automated deployment fails:

Deploy Infrastructure

az login --service-principal \
  --username $SERVICE_PRINCIPAL_ID \
  --password $SERVICE_PRINCIPAL_SECRET \
  --tenant $TENANT_ID

az deployment group create \
  --resource-group rg-glasslewis-prod \
  --template-file infra/main.bicep \
  --parameters @infra/params/prod.bicepparam

Deploy API

# Build API
dotnet publish src/GlassLewis.Api -c Release -o ./publish

# Deploy to Azure App Service
az webapp deploy \
  --resource-group rg-glasslewis-prod \
  --name app-glasslewis-api-prod \
  --src-path ./publish.zip

Deploy Frontend

# Build Angular app
cd clients/glasslewis.client.angular
npm ci
ng build --configuration production

# Deploy to Static Web App
az staticwebapp deploy \
  --name swa-glasslewis-client-prod \
  --app-location "dist/glasslewis-client" \
  --resource-group rg-glasslewis-prod

✅ Post-Deployment Verification

🔍 Functional Testing

Application loads successfully

curl -I https://your-production-domain.com
# Expected: HTTP 200 OK

Authentication flow works
- Login redirects to Azure Entra
- Successful authentication redirects back to app
- User can access protected resources

API endpoints respond correctly

# Test health endpoint
curl https://api.your-domain.com/health

# Test authenticated endpoint (with token)
curl -H "Authorization: Bearer $TOKEN" https://api.your-domain.com/api/v1/companies

Database connectivity verified

# Check application logs for database connections
az webapp log tail --name app-glasslewis-api-prod --resource-group rg-glasslewis-prod

📊 Performance Verification

Page load times acceptable (<3 seconds)

# Use curl to measure response times
curl -w "@curl-format.txt" -o /dev/null -s https://your-domain.com

API response times acceptable (<500ms)

# Test API performance
ab -n 100 -c 10 https://api.your-domain.com/health

Memory and CPU usage normal

# Monitor Azure metrics
az monitor metrics list \
  --resource /subscriptions/sub-id/resourceGroups/rg-glasslewis-prod/providers/Microsoft.Web/sites/app-glasslewis-api-prod \
  --metric "CpuPercentage,MemoryPercentage"

🔐 Security Verification

HTTPS enforced

curl -I http://your-domain.com
# Expected: 301 redirect to HTTPS

Security headers present

curl -I https://your-domain.com | grep -E "(Strict-Transport-Security|X-Content-Type-Options|X-Frame-Options)"

CORS policy working correctly

# Test CORS preflight
curl -H "Origin: https://unauthorized-domain.com" \
     -H "Access-Control-Request-Method: GET" \
     -X OPTIONS https://api.your-domain.com/api/v1/companies

Authentication tokens working
- Valid tokens accepted
- Invalid tokens rejected
- Token expiry handled correctly

📈 Monitoring Verification

Application Insights receiving data

# Check recent telemetry
az monitor app-insights events show \
  --app ai-glasslewis-prod \
  --resource-group rg-glasslewis-prod \
  --start-time "1 hour ago"

Health checks reporting green

curl https://api.your-domain.com/health
# Expected: {"status": "Healthy"}

Logs being written correctly

# Check application logs
az webapp log download \
  --name app-glasslewis-api-prod \
  --resource-group rg-glasslewis-prod \
  --log-file app-logs.zip

Alerts configured and functional

# Test alert by triggering condition (e.g., load test)
# Verify alert notifications received

🔧 Configuration Updates

📱 App Registration Updates

Update Client App Registration:

Azure Portal → App registrations → glasslewis-client-prod
Authentication → Add platform → Single-page application

Add redirect URIs:

https://your-production-domain.com
https://your-production-domain.com/auth/callback

Save changes

Update API App Registration:

Verify audience configuration matches production API
Update any application URIs if using custom domains

🌐 DNS and Domain Configuration

If using custom domain:

Configure CNAME records:

www.your-domain.com → your-app.azurestaticapps.net
api.your-domain.com → app-glasslewis-api-prod.azurewebsites.net

Configure SSL certificates in Azure
Update CORS origins in API configuration

🔧 Application Configuration

Verify production environment variables:

# In Azure App Service → Configuration → Application settings
AzureAd__ClientId=your-api-client-id
AzureAd__Authority=https://your-tenant.ciamlogin.com/
AzureAd__TenantId=your-tenant-id
ApplicationInsights__InstrumentationKey=your-app-insights-key

🚨 Rollback Procedures

When to Rollback

Immediate rollback if:

Authentication completely broken
Database connectivity issues
Critical functionality not working
Security vulnerabilities discovered
Performance degradation >50%

Rollback Steps

GitHub Actions Rollback:

# Redeploy previous successful release
git checkout v0.9.0  # Previous version
git tag -a v0.9.1 -m "Rollback to stable version"
git push origin v0.9.1

Manual Rollback:

# Redeploy previous API version
az webapp deployment slot swap \
  --name app-glasslewis-api-prod \
  --resource-group rg-glasslewis-prod \
  --slot staging

# Redeploy previous frontend version
az staticwebapp deploy \
  --name swa-glasslewis-client-prod \
  --app-location "dist/previous-version"

Database Rollback (if needed):

# Only if database migrations were deployed
dotnet ef migrations remove --connection "production-connection-string"

Verify Rollback:
- Application loads correctly
- Authentication works
- Critical functionality restored
- Performance back to normal

📋 Post-Deployment Tasks

📊 Monitoring Setup

Set up monitoring dashboards
- Application performance metrics
- Error rate tracking
- User activity monitoring

Configure alerting thresholds

# Example: High error rate alert
az monitor metrics alert create \
  --name "High Error Rate" \
  --resource-group rg-glasslewis-prod \
  --scopes "/subscriptions/sub-id/resourceGroups/rg-glasslewis-prod/providers/Microsoft.Web/sites/app-glasslewis-api-prod" \
  --condition "count 'Http 5xx' > 10" \
  --description "Alert when error rate exceeds threshold"

📚 Related Documentation

Azure Setup Guide - Infrastructure configuration
Common Issues - Troubleshooting guide