Security Incident Response - jra3/mulm GitHub Wiki

Security Incident Response

This guide provides procedures for responding to security incidents in the Mulm platform.

Overview

A security incident is any event that compromises the confidentiality, integrity, or availability of the system or user data.

Response Team:

  • System Administrator
  • Project Maintainer
  • Club Leadership (as needed)

Response Time:

  • Critical incidents: Immediate response (within 1 hour)
  • High severity: Response within 4 hours
  • Medium severity: Response within 24 hours
  • Low severity: Response within 1 week

Table of Contents

  1. Incident Classification
  2. Response Procedures
  3. Incident Types
  4. Communication
  5. Post-Incident Review

Incident Classification

Severity Levels

Severity Definition Examples Response Time
Critical Active attack, data breach, complete service outage SQL injection exploit, database leaked, site completely down Immediate (< 1 hour)
High Vulnerability discovered, partial outage, unauthorized access XSS vulnerability found, admin account compromised < 4 hours
Medium Security weakness, degraded service, suspicious activity Weak password policy, rate limiting bypass, unusual traffic < 24 hours
Low Minor security concern, no immediate risk Outdated dependency, information disclosure in logs < 1 week

Incident Types

Security Incidents:

  • Unauthorized access
  • Data breach or exposure
  • SQL injection exploit
  • XSS attack
  • CSRF attack
  • Session hijacking
  • DDoS attack
  • Malware/ransomware

Availability Incidents:

  • Complete service outage
  • Database corruption
  • Server compromise
  • Infrastructure failure

Data Integrity Incidents:

  • Database modification by unauthorized party
  • Data corruption
  • Accidental data deletion

Response Procedures

Phase 1: Detection & Assessment

1. Detect incident:

  • Automated monitoring alert
  • User report
  • Security researcher disclosure
  • Admin discovers during routine check

2. Initial assessment:

  • Confirm incident is real (not false positive)
  • Classify severity level
  • Identify affected systems
  • Determine if ongoing or contained

3. Document:

  • Create incident ticket (private GitHub issue or separate system)
  • Record detection time, detector, initial findings

Phase 2: Containment

Immediate actions (within 1 hour for critical):

For active attack:

# 1. Block attacking IP at nginx level
ssh BAP "sudo docker exec basny-nginx sh -c 'echo \"deny 1.2.3.4;\" >> /etc/nginx/conf.d/blocked-ips.conf'"
ssh BAP "sudo docker exec basny-nginx nginx -s reload"

# 2. Take site offline if needed (critical only)
ssh BAP "cd /opt/basny && sudo docker-compose -f docker-compose.prod.yml down"

# 3. Create database snapshot
aws --profile basny ec2 create-snapshot \
  --volume-id vol-0aba5b85a1582b2c0 \
  --description "Emergency snapshot - security incident $(date +%Y%m%d-%H%M%S)"

For vulnerability discovered:

# 1. Assess if being actively exploited
ssh BAP "sudo docker logs basny-app | grep -i 'suspicious-pattern'"

# 2. If not exploited, plan fix
# 3. If exploited, follow active attack procedures

For compromised credentials:

# 1. Revoke compromised credentials immediately
# - Rotate API keys (R2, OAuth, SMTP)
# - Reset admin passwords
# - Invalidate all sessions

# 2. Update credentials
ssh BAP "sudo vi /mnt/basny-data/app/config/config.production.json"

# 3. Restart app
ssh BAP "cd /opt/basny && sudo docker-compose -f docker-compose.prod.yml restart app"

Phase 3: Eradication

Fix the vulnerability:

# 1. Develop fix
# - Create patch
# - Write tests
# - Verify fix works

# 2. Deploy fix
ssh BAP "cd /opt/basny && git pull && sudo docker-compose -f docker-compose.prod.yml up -d --build"

# 3. Verify fix deployed
curl https://bap.basny.org/health

# 4. Test vulnerability is fixed
# - Attempt exploit (safely)
# - Verify it no longer works

Phase 4: Recovery

Restore normal operations:

# 1. Bring site back online (if taken down)
ssh BAP "cd /opt/basny && sudo docker-compose -f docker-compose.prod.yml up -d"

# 2. Monitor closely
ssh BAP "sudo docker logs basny-app -f"

# 3. Verify all functionality works
# - Test login
# - Test submissions
# - Test admin features

# 4. Check for data integrity
ssh BAP "sqlite3 /mnt/basny-data/app/database/database.db 'PRAGMA integrity_check;'"

Phase 5: Communication

Who to notify:

Audience When What to Say
Users If data breach or extended downtime Brief explanation, what we're doing, what they should do
Admins For any security incident Full details, actions taken, next steps
Security Researcher If they reported it Thank you, timeline for fix, credit offer
Authorities If required by law (data breach) As legally required

Example user notification:

Subject: BASNY BAP Security Update

On October 7, 2025, we discovered a security issue that may have
exposed member email addresses. We have:

✅ Fixed the vulnerability
✅ Verified no passwords were exposed
✅ Reset all active sessions as a precaution

What you should do:
- Log in again (your session was reset)
- Consider changing your password (optional, but recommended)
- Watch for phishing emails (we will never ask for your password)

We take security seriously and apologize for any concern this may cause.

If you have questions, please contact [email protected]

- BASNY BAP Team

Incident Types

SQL Injection Exploit

Detection:

  • Unusual database queries in logs
  • Unexpected data modifications
  • Error messages about SQL syntax

Response:

# 1. IMMEDIATE: Block attacking IP
ssh BAP "sudo docker exec basny-nginx sh -c 'echo \"deny ATTACKER_IP;\" >> /etc/nginx/conf.d/blocked-ips.conf'"
ssh BAP "sudo docker exec basny-nginx nginx -s reload"

# 2. Take snapshot
aws --profile basny ec2 create-snapshot --volume-id vol-0aba5b85a1582b2c0 --description "Pre-recovery snapshot"

# 3. Analyze attack
ssh BAP "sudo docker logs basny-app | grep 'ATTACKER_IP'"

# 4. Fix vulnerable code (use prepared statements)

# 5. Check for data tampering
ssh BAP "sqlite3 /mnt/basny-data/app/database/database.db 'SELECT * FROM members WHERE is_admin=1;'"
# Verify all admins are legitimate

# 6. Restore from backup if data was modified
# See Backup-Recovery guide

Data Breach

Detection:

  • Database file accessed by unauthorized party
  • Configuration file with secrets exposed
  • User reports receiving spam to unique email

Response:

# 1. CRITICAL: Determine what was exposed
# - Passwords? (hashed with scrypt - should be safe)
# - Emails? (notify users)
# - API keys? (rotate immediately)
# - Database file? (contains all above)

# 2. Rotate all secrets
# - R2 keys
# - OAuth secrets
# - SMTP password
# - Session secret

# 3. Invalidate all sessions
ssh BAP "sqlite3 /mnt/basny-data/app/database/database.db 'DELETE FROM sessions;'"

# 4. Force password reset for all users (if passwords may be compromised)
# - Send email with reset link to all members

# 5. Notify users
# - Email all members
# - Explain what happened
# - What data was exposed
# - What they should do

Compromised Admin Account

Detection:

  • Admin actions by user who claims they didn't do them
  • Suspicious admin actions in logs
  • Admin reports unauthorized access

Response:

# 1. IMMEDIATE: Disable account
ssh BAP "sqlite3 /mnt/basny-data/app/database/database.db \"UPDATE members SET is_admin=0 WHERE id=ADMIN_ID;\""

# 2. Delete all sessions for that admin
ssh BAP "sqlite3 /mnt/basny-data/app/database/database.db \"DELETE FROM sessions WHERE member_id=ADMIN_ID;\""

# 3. Review recent admin actions
ssh BAP "sqlite3 /mnt/basny-data/app/database/database.db \"
  SELECT * FROM submissions
  WHERE approved_by=ADMIN_ID
    AND approved_on > datetime('now', '-7 days')
  ORDER BY approved_on DESC;
\""

# 4. Check for malicious approvals or data changes

# 5. Reset admin password
# 6. Re-enable admin status after verification
# 7. Enable 2FA (if implemented)

DDoS Attack

Detection:

  • Extremely high traffic
  • Site slow or unresponsive
  • Rate limiter logs show excessive blocks

Response:

# 1. Check traffic sources
ssh BAP "sudo docker exec basny-nginx tail -1000 /var/log/nginx/access.log | awk '{print \$1}' | sort | uniq -c | sort -rn | head -20"

# 2. Block top offending IPs
ssh BAP "sudo docker exec basny-nginx sh -c 'echo \"deny 1.2.3.4;\" >> /etc/nginx/conf.d/blocked-ips.conf'"
ssh BAP "sudo docker exec basny-nginx nginx -s reload"

# 3. Enable Cloudflare (if not already)
# - Update DNS to point to Cloudflare
# - Enable "Under Attack Mode"

# 4. Lower rate limits temporarily
# Edit nginx.conf: rate=5r/s instead of 10r/s

# 5. Contact AWS support if attack is severe

Malware/Ransomware

Detection:

  • Files encrypted
  • Ransom note appears
  • Unusual file modifications

Response:

# 1. IMMEDIATELY disconnect from network
ssh BAP "sudo shutdown -h now"

# 2. DO NOT pay ransom

# 3. Recover from backups
# - Restore from EBS snapshot
# - See Backup-Recovery guide

# 4. Analyze how infection occurred
# - Review SSH access logs
# - Check for unauthorized users
# - Scan for backdoors

# 5. Rebuild from known-good state if necessary

Post-Incident Review

Within 1 Week After Resolution

Conduct post-mortem meeting:

  1. Timeline review

    • When was incident detected?
    • How long to contain?
    • How long to fully resolve?

  2. Root cause analysis

    • What caused the incident?
    • Why wasn't it prevented?
    • What gaps exist in security?

  3. Response effectiveness

    • What went well?
    • What went poorly?
    • What should we improve?

  4. Action items

    • Specific improvements to implement
    • Assign owners and deadlines
    • Update this playbook

Post-Incident Report Template

# Security Incident Report - [Date]

## Incident Summary
- **Date/Time:** October 7, 2025 14:30 UTC
- **Severity:** High
- **Type:** SQL Injection Attempt
- **Status:** Resolved

## Timeline
- 14:30 - Incident detected (error in logs)
- 14:35 - Confirmed as SQL injection attempt
- 14:40 - Blocked attacking IP
- 14:45 - Analyzed attack vector
- 15:00 - Deployed fix
- 15:30 - Verified fix works
- 16:00 - Monitoring resumed

## Impact
- No data was compromised (prepared statements prevented injection)
- No user impact (attack blocked)
- 30 minutes heightened monitoring

## Root Cause
- User-submitted search query not properly validated
- Query used prepared statements (safe), but should also validate input

## Response Actions
1. Blocked attacking IP: 1.2.3.4
2. Added input validation for search endpoint
3. Added test case for malicious input
4. Reviewed all other endpoints for similar issues

## Lessons Learned
- ✅ Prepared statements worked as designed (no injection)
- ✅ Monitoring detected attack quickly
- ⚠️ Input validation should be layered (not just SQL-safe)

## Follow-Up Actions
- [ ] Add input validation to all search endpoints
- [ ] Implement automated security scanning (SQLMap)
- [ ] Add rate limiting to search endpoints
- [ ] Update security documentation

## Recommendations
- All endpoints should validate input with Zod schemas
- Security scanning should run on every PR
- Consider Web Application Firewall (WAF)

Prevention

Proactive Measures

Regular security reviews:

  • Monthly dependency updates (npm audit)
  • Quarterly penetration testing
  • Annual security audit
  • Continuous monitoring

Security scanning:

# Check for known vulnerabilities
npm audit

# Check for secrets in code
git secrets --scan

# Static analysis
npm run lint

Training:

Emergency Contacts

Internal

Role Contact Responsibility
System Administrator [Contact info] Infrastructure, deployment, access
Project Maintainer [Contact info] Code, security fixes
Club President [Contact info] User communication, decisions

External

Service Contact Purpose
AWS Support Via AWS Console Infrastructure issues
Cloudflare Support Via dashboard R2 storage, DDoS protection
Google OAuth [email protected] OAuth issues

Recovery Procedures

Quick Recovery Checklist

For most incidents:

  • Take snapshot of current state
  • Analyze scope of incident
  • Contain the incident
  • Fix the vulnerability
  • Deploy the fix
  • Verify fix works
  • Monitor for recurrence
  • Document in incident report
  • Communicate to affected parties
  • Conduct post-mortem

Rollback Procedure

If recent deployment caused incident:

# 1. Check recent commits
ssh BAP "cd /opt/basny && git log --oneline -10"

# 2. Rollback to previous commit
ssh BAP "cd /opt/basny && git reset --hard HEAD~1"

# 3. Rebuild
ssh BAP "cd /opt/basny && sudo docker-compose -f docker-compose.prod.yml up -d --build"

# 4. Verify rollback fixed issue
curl https://bap.basny.org/health

Data Recovery

See: Backup & Recovery Guide for detailed procedures.

Legal Considerations

Data Breach Notification Laws

If personal data is compromised:

GDPR (if EU users):

  • Must notify supervisory authority within 72 hours
  • Must notify affected users "without undue delay"

US State Laws (vary by state):

  • Check requirements for New York (BASNY location)
  • May require notification to affected residents

Consult legal counsel if:

  • More than 500 users affected
  • Sensitive data exposed (SSN, payment info)
  • Cross-border data breach

Documentation Requirements

Keep records of:

  • When breach was discovered
  • What data was affected
  • How many users affected
  • What actions were taken
  • When users were notified

Testing Incident Response

Annual Security Drill

Simulate incident to test procedures:

  1. Plan drill scenario

    • Choose incident type (e.g., "SQL injection attempt")
    • Define scope and constraints
    • Schedule with team

  2. Execute drill

    • Run through response procedures
    • Time each phase
    • Document gaps or confusion

  3. Review results

    • What worked well?
    • What needs improvement?
    • Update procedures

  4. Improve

    • Update this playbook
    • Add missing tools
    • Train team on improvements

Related Documentation

Remember: Preparation is key. Review this guide periodically and conduct drills.

**Last Updated: November 2025