V2 CD Pipeline.md - 100-hours-a-week/KTB4-17th-wiki GitHub Wiki

CI/CD ์„ค๊ณ„

0. ๊ธฐ์กด CD ํŒŒ์ดํ”„๋ผ์ธ์˜ ํ•œ๊ณ„

V1์—์„œ๋Š” GitHub Actions๋ฅผ ํ†ตํ•ด ์ด๋ฏธ์ง€๋ฅผ ๋นŒ๋“œํ•œ ๋’ค EC2์— SSH๋กœ ์ ‘์†ํ•˜์—ฌ ๋ฐฐํฌ ์Šคํฌ๋ฆฝํŠธ๋ฅผ ์‹คํ–‰ํ•˜๊ณ , Nginx ๊ธฐ๋ฐ˜ Blue/Green ๋ฐฉ์‹์œผ๋กœ ๋ฐฐํฌํ•˜์˜€๋‹ค.

CD์˜ ํ•œ๊ณ„

V1์—์„œ๋Š” ์‹ ๊ทœ ์ปจํ…Œ์ด๋„ˆ๊ฐ€ Health Check๋ฅผ ํ†ต๊ณผํ•˜๋ฉด ์ •์ƒ ๋ฐฐํฌ๋กœ ํŒ๋‹จํ•˜๊ณ  ํŠธ๋ž˜ํ”ฝ์„ ์ „ํ™˜ํ•œ๋‹ค.

๊ทธ๋Ÿฌ๋‚˜ Health Check๋งŒ์œผ๋กœ๋Š” ํŠน์ • API ์˜ค๋ฅ˜, ์„œ๋น„์Šค ๊ฐ„ ์—ฐ๋™ ์‹คํŒจ ๋“ฑ ์‹ค์ œ ๊ธฐ๋Šฅ์˜ ์ •์ƒ ๋™์ž‘๊นŒ์ง€ ๋ณด์žฅํ•  ์ˆ˜ ์—†๋‹ค.

๋˜ํ•œ ๋ฐฐํฌ ์ดํ›„ ๋ฌธ์ œ๊ฐ€ ๋ฐœ์ƒํ•  ๊ฒฝ์šฐ ๊ฐœ๋ฐœ์ž๊ฐ€ ์ง์ ‘ ์žฅ์• ๋ฅผ ํ™•์ธํ•˜๊ณ  ์ด์ „ ๋ฒ„์ „์œผ๋กœ ๋กค๋ฐฑํ•ด์•ผ ํ•˜๋ฏ€๋กœ ์žฅ์•  ๊ฐ์ง€์™€ ๋ณต๊ตฌ ๊ณผ์ •์— ์ˆ˜๋™ ๊ฐœ์ž…์ด ํ•„์š”ํ•˜๋‹ค.

๋”ฐ๋ผ์„œ V2์—์„œ๋Š” ๋‹ค์Œ์„ ๋ชฉํ‘œ๋กœ CI/CD ํŒŒ์ดํ”„๋ผ์ธ์„ ๊ฐœ์„ ํ•œ๋‹ค.

  • Production ๋ฐฐํฌ ์ „ ์‹ค์ œ ํ™˜๊ฒฝ์—์„œ ํ•ต์‹ฌ ๊ธฐ๋Šฅ์„ ๊ฒ€์ฆํ•œ๋‹ค.
  • ๋ฐฐํฌ ์ค‘ Task๊ฐ€ ์ •์ƒ์ ์œผ๋กœ ์•ˆ์ •ํ™”๋˜๋Š”์ง€ ๊ฒ€์ฆํ•œ๋‹ค.
  • ๋ฐฐํฌ ์ดํ›„ ์‹ค์ œ ์„œ๋น„์Šค ์ง€ํ‘œ๋ฅผ ํ†ตํ•ด ์ด์ƒ ์—ฌ๋ถ€๋ฅผ ํŒ๋‹จํ•œ๋‹ค.
  • ๋ฐฐํฌ ์‹คํŒจ ์‹œ ์ด์ „ ์ •์ƒ Deployment๋กœ ์ž๋™ ๋กค๋ฐฑํ•œ๋‹ค.

1. CD ํŒŒ์ดํ”„๋ผ์ธ ์š”์•ฝ

2. ์ „์ฒด ๋ฐฐํฌ ์ „๋žต

V2์—์„œ๋Š” ๋ฐฐํฌ ๊ณผ์ •์—์„œ ๋‹ค์Œ๊ณผ ๊ฐ™์€ 3๋‹จ๊ณ„ ๊ฒ€์ฆ ๊ตฌ์กฐ๋ฅผ ์‚ฌ์šฉํ•œ๋‹ค.

๋‹จ๊ณ„ ๊ฒ€์ฆ ๋ชฉ์  ๊ฒ€์ฆ ๋ฐฉ๋ฒ• ์‹คํŒจ ์‹œ ์ฒ˜๋ฆฌ
Staging Smoke Test ์ฃผ์š” ๊ธฐ๋Šฅ์ด ์‹ค์ œ ํ™˜๊ฒฝ์—์„œ ์ •์ƒ ๋™์ž‘ํ•˜๋Š”๊ฐ€ Staging ECS + Staging DB์—์„œ ํ•ต์‹ฌ ์‹œ๋‚˜๋ฆฌ์˜ค ํ…Œ์ŠคํŠธ Production ๋ฐฐํฌ ์ค‘๋‹จ
๋ฐฐํฌ์ƒํƒœ ๊ฒ€์ฆ ์‹ ๊ทœ Task๊ฐ€ ์ •์ƒ์ ์œผ๋กœ ์‹คํ–‰๋˜๊ณ  ์•ˆ์ •ํ™”๋˜๋Š”๊ฐ€ ECS Deployment Circuit Breaker Deployment ์‹คํŒจ ๋ฐ ์ž๋™ ๋กค๋ฐฑ
Production ๊ฒ€์ฆ ์‹ค์ œ ํŠธ๋ž˜ํ”ฝ์—์„œ๋„ ์‹ ๊ทœ ๋ฒ„์ „์ด ์ •์ƒ ๋™์ž‘ํ•˜๋Š”๊ฐ€ CloudWatch Alarm Deployment ์‹คํŒจ ๋ฐ ์ž๋™ ๋กค๋ฐฑ

๊ฐ ๋‹จ๊ณ„๋Š” ์„œ๋กœ ๋‹ค๋ฅธ ๋ฌธ์ œ๋ฅผ ๊ฒ€์ฆํ•œ๋‹ค.

  • Staging Smoke Test: ๊ธฐ๋Šฅ ๋ฐ ์„œ๋น„์Šค ์—ฐ๋™ ๊ฒ€์ฆ
  • Circuit Breaker: ๋ฐฐํฌ ๋ฐ Task ์•ˆ์ •์„ฑ ๊ฒ€์ฆ
  • CloudWatch Alarm: ์‹ค์ œ ์šด์˜ ํŠธ๋ž˜ํ”ฝ์—์„œ ๋ฐœ์ƒํ•˜๋Š” ์ด์ƒ ๊ฒ€์ฆ

2-1. Staging ํ™˜๊ฒฝ ๊ตฌ์„ฑ

Production ๋ฐฐํฌ ์ „์— ์‹ค์ œ ECS ํ™˜๊ฒฝ์—์„œ ํ•ต์‹ฌ ์‚ฌ์šฉ์ž ์‹œ๋‚˜๋ฆฌ์˜ค๋ฅผ ๊ฒ€์ฆํ•˜๊ธฐ ์œ„ํ•ด Staging ํ™˜๊ฒฝ์„ ๊ตฌ์„ฑํ•œ๋‹ค.

Staging ํ™˜๊ฒฝ์€ ์ƒ์‹œ ์šด์˜ํ•˜์ง€ ์•Š๊ณ  ๋ฐฐํฌ ์‹œ์ ์—๋งŒ ํ™œ์„ฑํ™”ํ•œ๋‹ค.

์ฃผ์š” ์„ค์ •

ํ•ญ๋ชฉ ์„ค์ •
ECS Cluster Production๊ณผ ๋™์ผ Cluster ์‚ฌ์šฉ
ECS Service Staging ์ „์šฉ Service
ํ‰์ƒ์‹œ Desired Count 0
ํ…Œ์ŠคํŠธ ์‹œ Desired Count 1
DB Staging ์ „์šฉ DB (EC2 t3.micro)
ALB Target Group Staging ์ „์šฉ Target Group
ALB ๊ธฐ์กด ALB ๊ณต์œ 
์ ‘๊ทผ ๊ฒฝ๋กœ staging.star23.com
์‹คํ–‰ ์ฃผ์ฒด GitHub Actions
์ข…๋ฃŒ ์กฐ๊ฑด Smoke Test ์„ฑ๊ณต ๋˜๋Š” ์‹คํŒจ ํ›„ ํ•ญ์ƒ ์ข…๋ฃŒ

Staging ํ™˜๊ฒฝ์€ Production๊ณผ ECS Cluster ๋ฐ ALB๋ฅผ ๊ณต์œ ํ•˜์ง€๋งŒ Target Group, ECS Service, DB๋ฅผ ๋ถ„๋ฆฌํ•˜์—ฌ ํ…Œ์ŠคํŠธ๊ฐ€ Production ํ™˜๊ฒฝ์— ์˜ํ–ฅ์„ ์ฃผ์ง€ ์•Š๋„๋ก ํ•œ๋‹ค.


3. Production ๋ฐฐํฌ ์ „๋žต

3-1. Rolling Deployment ์ฃผ์š” ์„ค์ •

์„ค์ • ์ดˆ๊ธฐ ์„ค์ • ๋ชฉ์ 
Deployment Strategy Rolling Task๋ฅผ ์ ์ง„์ ์œผ๋กœ ์‹ ๊ทœ ๋ฒ„์ „์œผ๋กœ ๊ต์ฒด
minimumHealthyPercent 100% ๋ฐฐํฌ ์ค‘ ๊ธฐ์กด ์„œ๋น„์Šค ์ฒ˜๋ฆฌ ์šฉ๋Ÿ‰ ์œ ์ง€
maximumPercent 200% ๊ธฐ์กด Task๋ฅผ ์œ ์ง€ํ•˜๋ฉด์„œ ์‹ ๊ทœ Task ์šฐ์„  ์‹คํ–‰
Deployment Circuit Breaker Enable ์‹ ๊ทœ Deployment ์‹คํŒจ ๊ฐ์ง€
Circuit Breaker Rollback Enable ์‹คํŒจ ์‹œ ์ด์ „ ์ •์ƒ Deployment ๋ณต๊ตฌ

3-2. Deployment Circuit Breaker

Rolling Deployment ๊ณผ์ •์—์„œ ์‹ ๊ทœ Task๊ฐ€ ์ •์ƒ์ ์œผ๋กœ ์‹คํ–‰๋˜์ง€ ๋ชปํ•˜๋Š” ๊ฒฝ์šฐ๋ฅผ ์ž๋™์œผ๋กœ ๊ฐ์ง€ํ•˜๊ธฐ ์œ„ํ•ด ECS Deployment Circuit Breaker๋ฅผ ํ™œ์„ฑํ™”ํ•œ๋‹ค.

Circuit Breaker๋ฅผ ํ†ตํ•ด ์‹ ๊ทœ Deployment๊ฐ€ ์ •์ƒ ์ƒํƒœ์— ๋„๋‹ฌํ•˜์ง€ ๋ชปํ•˜๋Š” ์ƒํ™ฉ์„ ๊ฐ์ง€ํ•˜์—ฌ Deployment๋ฅผ ์‹คํŒจ ์ฒ˜๋ฆฌํ•œ๋‹ค.

์ฃผ์š” ์„ค์ •

ํ•ญ๋ชฉ ์„ค์ •
Circuit Breaker Enable
Rollback Enable
์ ์šฉ ๋Œ€์ƒ Production ECS Service
์‹คํŒจ ๊ฒฐ๊ณผ Deployment FAILED
๋ณต๊ตฌ ๋Œ€์ƒ ์ด์ „ ์ •์ƒ COMPLETED Deployment

์ด๋ฅผ ํ†ตํ•ด ์‹ ๊ทœ Task๊ฐ€ ์ •์ƒ์ ์œผ๋กœ ๋ฐฐํฌ๋˜์ง€ ๋ชปํ•˜๋Š” ๊ฒฝ์šฐ ECS๊ฐ€ ์ด์ „ ์ •์ƒ Deployment๋กœ ์ž๋™ ๋ณต๊ตฌํ•œ๋‹ค.

3-3. Production ๋ฐฐํฌ ์ดํ›„ ๊ฒ€์ฆ

Task๊ฐ€ ์ •์ƒ์ ์œผ๋กœ ์‹คํ–‰๋˜๊ณ  Health Check๋ฅผ ํ†ต๊ณผํ–ˆ๋‹ค๊ณ  ํ•ด์„œ ์‹ค์ œ ์„œ๋น„์Šค์˜ ๋ชจ๋“  ๊ธฐ๋Šฅ์ด ์ •์ƒ์ž„์„ ๋ณด์žฅํ•  ์ˆ˜๋Š” ์—†๋‹ค.

๋”ฐ๋ผ์„œ Production์— ์‹ค์ œ ํŠธ๋ž˜ํ”ฝ์ด ์œ ์ž…๋œ ์ดํ›„ CloudWatch Alarm์„ ์ด์šฉํ•˜์—ฌ ์„œ๋น„์Šค ์ƒํƒœ๋ฅผ ์ถ”๊ฐ€๋กœ ๊ฒ€์ฆํ•œ๋‹ค.

3-3-1. ์ดˆ๊ธฐ ๋ชจ๋‹ˆํ„ฐ๋ง ์ง€ํ‘œ

์ง€ํ‘œ ๋ชฉ์  ๊ธฐ์ค€
ALB Target 5XX ์‹ ๊ทœ Application ์˜ค๋ฅ˜ ํƒ์ง€ SLO ์„ ์ • ํ›„ ๊ฒฐ์ •
Target Response Time ์‘๋‹ต ์„ฑ๋Šฅ ์ €ํ•˜ ํƒ์ง€ SLO ์„ ์ • ํ›„ ๊ฒฐ์ •

3-3-2. CloudWatch Alarm๊ณผ ์ž๋™ ๋กค๋ฐฑ

Production ECS Deployment์— CloudWatch Alarm์„ ์—ฐ๊ฒฐํ•œ๋‹ค.

์‹ ๊ทœ ๋ฒ„์ „ ๋ฐฐํฌ ๊ณผ์ •์—์„œ ์ง€์ •๋œ CloudWatch Alarm์ด ALARM ์ƒํƒœ๊ฐ€ ๋˜๋ฉด ํ•ด๋‹น Deployment๋ฅผ ์‹คํŒจ๋กœ ํŒ๋‹จํ•˜๊ณ  ์ž๋™ ๋กค๋ฐฑํ•œ๋‹ค.

์ฃผ์š” ์„ค์ •

ํ•ญ๋ชฉ ์„ค์ •
Deployment Alarm Enable
Rollback Enable
Primary Metric ALB Target 5XX
Latency Metric Target Response Time
Bake Time ์ดˆ๊ธฐ๊ฐ’ ๊ฒ€์ฆ ํ•„์š”
Alarm ๋ฐœ์ƒ ์‹œ Deployment FAILED
Rollback ๋Œ€์ƒ Last COMPLETED Deployment`

์ „์ฒด ํ๋ฆ„์€ ๋‹ค์Œ๊ณผ ๊ฐ™๋‹ค.

Rolling Deployment ์™„๋ฃŒ 
โ†’ ์‹ค์ œ ํŠธ๋ž˜ํ”ฝ ์ฒ˜๋ฆฌ 
โ†’ CloudWatch Alarm ๊ด€์ฐฐ 
โ†’ ์ •์ƒ ์‹œ COMPLETED / ์ด์ƒ ์‹œ FAILED 
โ†’ ์ด์ƒ ์‹œ Automatic Rollback

์‹ค์ œ ์„œ๋น„์Šค ์ง€ํ‘œ๋ฅผ ๊ด€์ฐฐํ•จ์œผ๋กœ์จ Health Check๋งŒ์œผ๋กœ ๋ฐฐํฌ ์„ฑ๊ณต ์—ฌ๋ถ€๋ฅผ ๊ฒฐ์ •ํ–ˆ๋˜ V1์˜ ํ•œ๊ณ„๋ฅผ ๋ณด์™„ํ•œ๋‹ค.


4. ์ตœ์ข… ๋ฐฐํฌ ์„ฑ๊ณต ๊ธฐ์ค€

V1์—์„œ๋Š” Health Check ํ†ต๊ณผ ์—ฌ๋ถ€๋งŒ์œผ๋กœ ๋ฐฐํฌ ์„ฑ๊ณต์„ ํŒ๋‹จํ•˜์˜€๋‹ค.

V2์—์„œ๋Š” ๋ฐฐํฌ ์„ฑ๊ณต์„ ๋‹ค์Œ ์„ธ ๋‹จ๊ณ„๋กœ ํ™•์žฅํ•œ๋‹ค.

  1. Staging Smoke Test ํ†ต๊ณผ
    • ์‹ค์ œ ๋ฐฐํฌ ํ™˜๊ฒฝ๊ณผ ์œ ์‚ฌํ•œ ํ™˜๊ฒฝ์—์„œ ํ•ต์‹ฌ ๊ธฐ๋Šฅ์ด ์ •์ƒ์ ์œผ๋กœ ๋™์ž‘ํ•œ๋‹ค.
  2. Production Deployment Steady State ๋„๋‹ฌ
    • ์‹ ๊ทœ Task๊ฐ€ ์ •์ƒ์ ์œผ๋กœ ์‹คํ–‰๋˜๊ณ  ECS Deployment๊ฐ€ Steady State์— ๋„๋‹ฌํ•œ๋‹ค.
  3. Production ์ง€ํ‘œ ์ •์ƒ
    • ์‹ค์ œ ์‚ฌ์šฉ์ž ํŠธ๋ž˜ํ”ฝ์„ ์ฒ˜๋ฆฌํ•˜๋Š” ๋™์•ˆ ์ง€์ •๋œ CloudWatch Alarm์ด ๋ฐœ์ƒํ•˜์ง€ ์•Š๋Š”๋‹ค.

์„ธ ์กฐ๊ฑด์„ ๋ชจ๋‘ ๋งŒ์กฑํ•œ ๊ฒฝ์šฐ์—๋งŒ Production Deployment๋ฅผ ์ตœ์ข… ์„ฑ๊ณต์œผ๋กœ ํŒ๋‹จํ•œ๋‹ค.

์ด๋ฅผ ํ†ตํ•ด V1์˜ ๋‹จ์ˆœ Health Check ๊ธฐ๋ฐ˜ ๋ฐฐํฌ ๊ฒ€์ฆ๊ณผ ์ˆ˜๋™ ๋กค๋ฐฑ ๊ตฌ์กฐ๋ฅผ ๊ฐœ์„ ํ•˜๊ณ , ๋ฐฐํฌ ์ „ ๊ธฐ๋Šฅ ๊ฒ€์ฆ๋ถ€ํ„ฐ ๋ฐฐํฌ ํ›„ ์šด์˜ ์ง€ํ‘œ ๊ฒ€์ฆ ๋ฐ ์ž๋™ ๋ณต๊ตฌ๊นŒ์ง€ ํฌํ•จํ•˜๋Š” CD ํŒŒ์ดํ”„๋ผ์ธ์„ ๊ตฌ์„ฑํ•œ๋‹ค.


5. ๊ฐœ์„ ์  ์ด ์ •๋ฆฌ

๊ตฌ๋ถ„ V1 V2 ๊ฐœ์„ ์ 
๋ฐฐํฌ ๋ฐฉ์‹ Nginx ๊ธฐ๋ฐ˜ Blue/Green ECS Rolling Deployment ECS Service ๊ธฐ๋ฐ˜์˜ ์ ์ง„์  Task ๊ต์ฒด
๋ฐฐํฌ ์ค‘ ๊ฐ€์šฉ์„ฑ ์ผ์ • ์‹œ๊ฐ„ Blue/Green ํ™˜๊ฒฝ ์œ ์ง€ minimumHealthyPercent๋กœ ์ตœ์†Œ Healthy Task ์œ ์ง€ Rolling ์ค‘ ์„œ๋น„์Šค ์ฒ˜๋ฆฌ ์šฉ๋Ÿ‰ ์œ ์ง€
๋ฐฐํฌ ์ž์› ์ œ์–ด Blue/Green ์ปจํ…Œ์ด๋„ˆ ์ง์ ‘ ๊ด€๋ฆฌ maximumPercent๋กœ ์ตœ๋Œ€ Task ์ˆ˜ ์ œ์–ด ๋ฐฐํฌ ์ค‘ ์ถ”๊ฐ€ ์ž์› ์‚ฌ์šฉ๋Ÿ‰ ์ œ์–ด
๋ฐฐํฌ ์•ˆ์ •์„ฑ ํŒ๋‹จ Health Check Stag ํ™˜๊ฒฝ Smoke Test, Steady State ์‹ ๊ทœ Task์˜ ๊ธฐ๋™ ๋ฐ ๋ฐฐํฌ ์•ˆ์ •ํ™” ์‹คํŒจ ์ž๋™ ๊ฐ์ง€
๋ฐฐํฌ ํ›„ ๊ฒ€์ฆ ๋ณ„๋„ ๊ฒ€์ฆ ์—†์Œ CloudWatch Alarm์œผ๋กœ 5XX, Response Time ๊ด€์ฐฐ ์‹ค์ œ ์‚ฌ์šฉ์ž ํŠธ๋ž˜ํ”ฝ ๊ธฐ๋ฐ˜ ์ด์ƒ ํƒ์ง€
๋ฐฐํฌ ์„ฑ๊ณต ๊ธฐ์ค€ Health Check ํ†ต๊ณผ Steady State ๋„๋‹ฌ + ์ผ์ • ์‹œ๊ฐ„ ๋™์•ˆ Alarm ๋ฏธ๋ฐœ์ƒ ๋‹จ์ˆœ ๊ธฐ๋™ ์—ฌ๋ถ€์—์„œ ์‹ค์ œ ์„œ๋น„์Šค ์ƒํƒœ๊นŒ์ง€ ๊ฒ€์ฆ ๋ฒ”์œ„ ํ™•๋Œ€
์žฅ์•  ๊ฐ์ง€ ๊ฐœ๋ฐœ์ž๊ฐ€ ์ง์ ‘ ํ™•์ธ Circuit Breaker + CloudWatch Alarm ๋ฐฐํฌ ๊ณผ์ •์˜ ์žฅ์•  ์ž๋™ ๊ฐ์ง€
Rollback ์‹คํ–‰ EC2 ์ ‘์† ํ›„ ์ˆ˜๋™ ๋ณต๊ตฌ Automatic Rollback ์žฅ์•  ๋ฐœ์ƒ ์‹œ ์ด์ „ ์ •์ƒ Deployment๋กœ ์ž๋™ ๋ณต๊ตฌ
Rollback ๋Œ€์ƒ ๊ฐœ๋ฐœ์ž๊ฐ€ ์ด์ „ ์ด๋ฏธ์ง€/์ปจํ…Œ์ด๋„ˆ ๋ฒ„์ „ ํ™•์ธ Last COMPLETED Deployment ๋ณต๊ตฌ ๊ณผ์ • ์ž๋™ํ™”
์šด์˜ ๊ด€๋ฆฌ SSH ๋ฐ ๋ฐฐํฌ ์Šคํฌ๋ฆฝํŠธ ์ค‘์‹ฌ ECS Service ์ค‘์‹ฌ ๊ด€๋ฆฌ ์ง€์  ๊ฐ์†Œ