402 V1 load test execution - 100-hours-a-week/KTB4-3rd-CLOUD GitHub Wiki

V1 ๋ถ€ํ•˜ํ…Œ์ŠคํŠธ ์‹คํ–‰ ๋ฌธ์„œ

1. ๋ชฉ์ ๊ณผ ๋ฒ”์œ„

์ด ๋ฌธ์„œ๋Š” MAU 1,150,000๋ช…์„ ๊ธฐ์ค€์œผ๋กœ V1์˜ ์ง€์† ์ฒ˜๋ฆฌ๋Ÿ‰๊ณผ ๋ณ‘๋ชฉ ์ง€์ ์„ ์ˆ˜์น˜ํ™”ํ•˜๊ณ , ๊ทธ ๊ฒฐ๊ณผ๋กœ ์ˆ˜์ง ํ™•์žฅยทDB ๋ถ„๋ฆฌยท์™ธ๋ถ€ ํ˜ธ์ถœ ๊ฒฉ๋ฆฌยทWebSocket ๋ถ„๋ฆฌ ์—ฌ๋ถ€๋ฅผ ๊ฒฐ์ •ํ•˜๊ธฐ ์œ„ํ•œ ์‹คํ–‰ ๊ธฐ์ค€์ด๋‹ค.

์ธก์ • ๋Œ€์ƒ์€ ๋กœ์ปฌ์—์„œ ์‹คํ–‰ํ•œ k6๊ฐ€ V1์˜ ๊ณต๊ฐœ ์ง„์ž…์ ์œผ๋กœ ๋ณด๋‚ธ ํŠธ๋ž˜ํ”ฝ์ด๋‹ค. ์ฆ‰, ๋ถ€ํ•˜ ๊ฒฝ๋กœ์— public DNS/EIP, Internet routing, TLS, Nginx๋ฅผ ํฌํ•จํ•œ๋‹ค. V1 private IP๋Š” ๋ถ€ํ•˜ ๋Œ€์ƒ์ด ์•„๋‹ˆ๋ผ Prometheus๊ฐ€ exporter๋ฅผ ์ˆ˜์ง‘ํ•˜๋Š” ์ฃผ์†Œ๋กœ๋งŒ ์‚ฌ์šฉํ•œ๋‹ค.

๊ธฐ์กด perf/ ๋ถ€ํ•˜ํ…Œ์ŠคํŠธ ์ฝ”๋“œ๋Š” ์‚ฌ์šฉํ•˜์ง€ ์•Š๋Š”๋‹ค. ์ƒˆ ์ฝ”๋“œ๋Š” loadtest-scenarios/์— ๋…๋ฆฝ์ ์œผ๋กœ ์ž‘์„ฑ๋˜์–ด ์žˆ๋‹ค.

ํŒ์ • ์ •์˜

C_knee = SLO ๋˜๋Š” ๋ฐ์ดํ„ฐ ์ •ํ•ฉ์„ฑ์ด ์ฒ˜์Œ ๊นจ์ง€๋Š” ์ตœ๋Œ€ ์ง€์† ๋ถ€ํ•˜
C_safe = C_knee ร— 0.7

์‘๋‹ต์‹œ๊ฐ„ยท์˜ค๋ฅ˜์œจยท์ •ํ•ฉ์„ฑ์„ 1์ฐจ ํŒ์ •๊ฐ’์œผ๋กœ ์‚ฌ์šฉํ•˜๊ณ  CPU, GC, HikariCP, MySQL lock, ๋””์Šคํฌ์™€ ๋„คํŠธ์›Œํฌ ์ง€ํ‘œ๋Š” ์›์ธ ๋ถ„์„์šฉ ์ฆ๊ฑฐ๋กœ ์‚ฌ์šฉํ•œ๋‹ค.

2. MAUยทDAUยทRPS ์žฌ์‚ฐ์ •

2.1 ๊ณ ์ •๊ฐ’๊ณผ ๋ฒค์น˜๋งˆํฌ์˜ ์—ญํ• 

๋ชจ์—ฌํƒ€ MAU๋Š” 1,150,000๋ช…์œผ๋กœ ๊ณ ์ •ํ•œ๋‹ค. BlaBlaCar ์ˆ˜์น˜๋Š” ๋ชจ์—ฌํƒ€์˜ MAU๋ฅผ ๋‹ค์‹œ ๊ณ„์‚ฐํ•˜๋Š” ์นจํˆฌ์œจ์ด ์•„๋‹ˆ๋ผ, ์šด์˜ ๋กœ๊ทธ๊ฐ€ ์—†์„ ๋•Œ DAU/MAU ํ™œ๋™๋ฅ ๊ณผ ์„ธ์…˜ ๋นˆ๋„๋ฅผ ์ •ํ•˜๋Š” ์ฐธ๊ณ ๊ฐ’์œผ๋กœ๋งŒ ์‚ฌ์šฉํ•œ๋‹ค.

BlaBlaCar ์ŠคํŽ˜์ธ 2025 ๊ณต์‹ ์ž๋ฃŒ์—๋Š” ๋‹ค์Œ ๊ฐ’์ด ๊ณต๊ฐœ๋˜์–ด ์žˆ๋‹ค.

  • ์›”ํ‰๊ท  ๊ณ ์œ  ํ™œ์„ฑ ์‚ฌ์šฉ์ž: 300,000~400,000๋ช…
  • ์—ฐ๊ฐ„ ์นดํ’€ ์Šน๊ฐ: 4,000,000๋ช…

์ถœ์ฒ˜: BlaBlaCar Spain 2025 activity report

์—ฐ๊ฐ„ ์Šน๊ฐ ์ˆ˜๋ฅผ ์ผ๊ฐ„ ํ™œ๋™ ํ”„๋ก์‹œ๋กœ ๋ณ€ํ™˜ํ•œ๋‹ค.

์ผ ํ‰๊ท  ์Šน๊ฐ ํ™œ๋™ = 4,000,000 รท 365 = 10,959๋ช…/day

DAU/MAU ํ•˜๋‹จ = 10,959 รท 400,000 = 2.740%
DAU/MAU ์ค‘์•™ = 10,959 รท 350,000 = 3.131%
DAU/MAU ์ƒ๋‹จ = 10,959 รท 300,000 = 3.653%

๋”ฐ๋ผ์„œ ๊ณ ์ • MAU์— ์ ์šฉํ•œ DAU ๋ฒ”์œ„๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™๋‹ค.

๊ตฌ๋ถ„ ํ™œ๋™๋ฅ  ํ”„๋ก์‹œ ๊ณ„์‚ฐ ์ถ”์ • DAU
ํ•˜๋‹จ 2.740% 1,150,000 ร— 2.740% 31,507
์ค‘์•™ 3.131% 1,150,000 ร— 3.131% 36,008
์ƒ๋‹จ 3.653% 1,150,000 ร— 3.653% 42,009

์ด ๊ฐ’์€ ๊ณ ์œ  ๋กœ๊ทธ์ธ DAU๊ฐ€ ์•„๋‹ˆ๋‹ค. ํ•˜๋ฃจ ๋ณต์ˆ˜ ์Šน๊ฐ ํ™œ๋™์ด ํ•ฉ์ณ์ง€๊ฑฐ๋‚˜ ์šด์ „์žยท๊ฒ€์ƒ‰๋งŒ ํ•œ ์‚ฌ์šฉ์ž๊ฐ€ ๋น ์งˆ ์ˆ˜ ์žˆ์œผ๋ฏ€๋กœ, ์‹ค์ œ ๋กœ๊ทธ๊ฐ€ ํ™•๋ณด๋˜๋ฉด ๊ณ ์œ  DAUยท์„ธ์…˜ ์ˆ˜ยท์„ธ์…˜๋‹น ์š”์ฒญ ์ˆ˜๋กœ ๊ต์ฒดํ•œ๋‹ค.

2.2 ๊ธฐ์ค€ ํ˜ผํ•ฉ RPS

ํ˜„์žฌ ์‹œํ€€์Šค์˜ ํ•œ ์„ธ์…˜์„ HTTP 13๊ฑด์œผ๋กœ ์ •์˜ํ•œ๋‹ค.

๋ชฉ๋กยท๊ฒ€์ƒ‰ 4๊ฑด + ์ƒ์„ธ 4๊ฑด + ํ˜ผ์žก๋„ 3๊ฑด + ์“ฐ๊ธฐ 1๊ฑด + ์ธ์ฆยท๊ธฐํƒ€ 1๊ฑด
= ์„ธ์…˜๋‹น HTTP 13๊ฑด

ํ”ผํฌ 4์‹œ๊ฐ„์— ์ผ๊ฐ„ ์„ธ์…˜์˜ 70%๊ฐ€ ์ง‘์ค‘๋˜๊ณ , ๊ทธ ์•ˆ์—์„œ ์ˆœ๊ฐ„ ํ”ผํฌ ๊ณ„์ˆ˜ 2.0์ด ์ ์šฉ๋œ๋‹ค๊ณ  ๊ฐ€์ •ํ•œ๋‹ค.

ํ”ผํฌ ์„ธ์…˜ ์‹œ์ž‘๋ฅ 
= DAU ร— (์ผ ์„ธ์…˜/DAU) ร— 70% รท (4 ร— 3,600์ดˆ) ร— 2.0

ํ”ผํฌ HTTP RPS
= ํ”ผํฌ ์„ธ์…˜ ์‹œ์ž‘๋ฅ  ร— 13๊ฑด

์ผ ์„ธ์…˜/DAU๋ฅผ 1ํšŒ๋กœ ๋‘” ๊ฒฐ๊ณผ๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™๋‹ค.

๊ตฌ๋ถ„ DAU ํ”ผํฌ ์„ธ์…˜ ์‹œ์ž‘๋ฅ  ๊ณ„์‚ฐ HTTP RPS
ํ•˜๋‹จ 31,507 3.063/s 39.82
์ค‘์•™ 36,008 3.501/s 45.51
์ƒ๋‹จ 42,009 4.084/s 53.09

๋”ฐ๋ผ์„œ ์šด์˜ ๋ฐ์ดํ„ฐ๊ฐ€ ์—†๋Š” ์ฒซ ๊ธฐ์ค€ ๋ถ€ํ•˜๋Š” 50 RPS๋กœ ๊ณ ์ •ํ•œ๋‹ค. ์ค‘์•™๊ฐ’ 45.51 RPS์—์„œ์˜ ์ž‘์—…๋ณ„ ๋ถ„๋ฐฐ๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™๋‹ค.

์ž‘์—… ์„ธ์…˜ ๋น„์œจ ์ค‘์•™ RPS
๋ชฉ๋กยท๊ฒ€์ƒ‰ 4/13 14.00
์ƒ์„ธ 4/13 14.00
ํ˜ผ์žก๋„ 3/13 10.50
์“ฐ๊ธฐ 1/13 3.50
์ธ์ฆยท๊ธฐํƒ€ 1/13 3.50

์„ธ์…˜์ด ํ•˜๋ฃจ 2ํšŒยท3ํšŒ๋ผ๋ฉด RPS๋Š” ์„ ํ˜•์œผ๋กœ ์•ฝ 91ยท137 RPS๊ฐ€ ๋œ๋‹ค. ์ด๋Š” ํ•œ๊ณ„๊ฐ’์ด ์•„๋‹ˆ๋ผ ๋ฏผ๊ฐ๋„ ํ™•์ธ๊ฐ’์ด๋‹ค.

2.3 ์šฉ๋Ÿ‰ ํƒ์ƒ‰ ๋‹จ๊ณ„

๊ธฐ์ค€๊ฐ’ ์žฌํ˜„ ํ›„์—๋Š” ๋‹ค์Œ ์ˆœ์„œ๋กœ ํ•œ ๋‹จ๊ณ„์”ฉ ์˜ฌ๋ฆฐ๋‹ค.

25 โ†’ 50 โ†’ 75 โ†’ 100 โ†’ 150 โ†’ 200 โ†’ 300 โ†’ 400 โ†’ 512 RPS

๊ฐ ๋‹จ๊ณ„๋Š” ์›Œ๋ฐ์—… 5๋ถ„, ์ธก์ • 10๋ถ„, ๋™์ผ ์กฐ๊ฑด 3ํšŒ ๋ฐ˜๋ณต์„ ๊ธฐ๋ณธ์œผ๋กœ ํ•œ๋‹ค. ์ตœ์ดˆ SLO ์œ„๋ฐ˜ ๋‹จ๊ณ„์˜ ์•ž๋’ค๋ฅผ 10~20% ๊ฐ„๊ฒฉ์œผ๋กœ ์žฌ์ธก์ •ํ•ด C_knee๋ฅผ ์ขํžŒ๋‹ค.

2.4 ํ–‰์‚ฌ burst

ํ–‰์‚ฌ ์‚ฌ์šฉ์ž ์ˆ˜๋Š” DAU์˜ 10%, ์ด๋ฒคํŠธ ์„ธ์…˜์€ ์‚ฌ์šฉ์ž๋‹น HTTP 15๊ฑด, ๊ทธ์ค‘ 50%๊ฐ€ WebSocket์— ์ ‘์†ํ•˜๊ณ  ์ ‘์†์ž๋‹น 10๋ถ„ ๋™์•ˆ ๋ฉ”์‹œ์ง€ 6๊ฑด์„ ๋ณด๋‚ธ๋‹ค๋Š” ๊ฐ€์ •์ด๋‹ค.

0 โ†’ 5๋ถ„ ๋™์•ˆ peak๊นŒ์ง€ ์ƒ์Šน โ†’ 5๋ถ„ ๋™์•ˆ 0๊นŒ์ง€ ํ•˜๊ฐ•
HTTP peak RPS = 2 ร— ์ด๋ฒคํŠธ ์‚ฌ์šฉ์ž ร— 15 รท 600์ดˆ
๊ตฌ๋ถ„ ์ด๋ฒคํŠธ ์‚ฌ์šฉ์ž HTTP peak ์ตœ์ข… WS ์—ฐ๊ฒฐ ์—ฐ๊ฒฐ๋ฅ  inbound ๋ฉ”์‹œ์ง€
ํ•˜๋‹จ 3,151 157.5 RPS 1,575 5.25/s 31.5/s
์ค‘์•™ 3,601 180.0 RPS 1,800 6.00/s 36.0/s
์ƒ๋‹จ 4,201 210.0 RPS 2,100 7.00/s 42.0/s

์˜ˆ์ƒ burst๋Š” 160 โ†’ 180 โ†’ 210 RPS, ํƒ์ƒ‰ burst๋Š” 250 โ†’ 300 โ†’ 400 โ†’ 512 RPS๋กœ ์‹คํ–‰ํ•œ๋‹ค. WebSocket์€ ์—ฐ๊ฒฐ ์ˆ˜์™€ ๋ฉ”์‹œ์ง€ ์ˆ˜๋ฅผ HTTP์™€ ๋ณ„๋„๋กœ ์ฆ๊ฐ€์‹œํ‚จ๋‹ค.

3. ๋ถ€ํ•˜ํ…Œ์ŠคํŠธ ํ™˜๊ฒฝ ์„ค์ •

3.1 ์‹คํ–‰ ํ† ํด๋กœ์ง€

๋กœ์ปฌ k6 HTTP ํ”„๋กœ์„ธ์Šค โ”€โ”
                       โ”œโ”€ Public DNS/EIP โ†’ TLS โ†’ Nginx โ†’ Spring โ”€โ”ฌโ”€ MySQL
๋กœ์ปฌ k6 WS ํ”„๋กœ์„ธ์Šค โ”€โ”€โ”€โ”˜                                          โ””โ”€ FastAPI/์™ธ๋ถ€ API

V1 EC2 โ”€โ”€ Node Exporter / cAdvisor / Spring metrics / MySQL exporter / Nginx exporter
             โ”‚
             โ””โ”€ private scrape โ”€โ”€ ์ž„์‹œ Prometheus + Grafana EC2
  • ๋ถ€ํ•˜ ๋ฐœ์ƒ๊ธฐ๋Š” EC2์— ๋งŒ๋“ค์ง€ ์•Š๊ณ  ๋กœ์ปฌ์—์„œ ์‹คํ–‰ํ•œ๋‹ค.
  • HTTP์™€ WebSocket์€ ๋กœ์ปฌ์—์„œ ๋ณ„๋„ ํ”„๋กœ์„ธ์Šค๋กœ ์‹คํ–‰ํ•œ๋‹ค.
  • BASE_URL์€ https://, WS_URL์€ wss:// public ์ฃผ์†Œ๋งŒ ์‚ฌ์šฉํ•œ๋‹ค.
  • target_private_ip๋Š” Prometheus scrape ์ „์šฉ์ด๋ฉฐ k6 ๋Œ€์ƒ์— ๋„ฃ์ง€ ์•Š๋Š”๋‹ค.
  • ๋กœ์ปฌ CPU 60% ๋ฏธ๋งŒ, swap 0, ์—…๋กœ๋“œ ๋Œ€์—ญํญ 70% ๋ฏธ๋งŒ, dropped iterations 0, ๋ชฉํ‘œ ์ „๋‹ฌ๋ฅ  95% ์ด์ƒ์„ ์œ ์ง€ํ•œ๋‹ค.

3.2 Terraform ์Šคํƒ์˜ ์ฑ…์ž„

loadtest-infra/๋Š” ๊ธฐ์กด infra/์™€ ๋ณ„๋„ state๋ฅผ ์‚ฌ์šฉํ•œ๋‹ค. ๋‹ค์Œ๋งŒ ๊ด€๋ฆฌํ•œ๋‹ค.

  • ์ž„์‹œ Prometheus/Grafana EC2 ๋ฐ EIP
  • Prometheus ๊ฒฐ๊ณผ ๋ณด๊ด€์šฉ private S3 bucket
  • monitoring security group
  • ๊ธฐ์กด V1 SG์— exporter port๋ฅผ ์ž„์‹œ๋กœ ํ—ˆ์šฉํ•˜๋Š” ingress rule
  • ๊ธฐ์กด V1 EC2์— exporter๋ฅผ ์„ค์น˜ยท์ œ๊ฑฐํ•˜๋Š” SSM Command document

์ด ์Šคํƒ์€ ์• ํ”Œ๋ฆฌ์ผ€์ด์…˜ ์ปจํ…Œ์ด๋„ˆ๋ฅผ ์ƒ์„ฑํ•˜์ง€ ์•Š๋Š”๋‹ค. ๋Œ€์ƒ ์• ํ”Œ๋ฆฌ์ผ€์ด์…˜์€ ๊ธฐ์กด V1 ๋ฐฐํฌ๋ฅผ ์‚ฌ์šฉํ•ด์•ผ ํ•œ๋‹ค. ํ˜„์žฌ V1 ์ด๋ฏธ์ง€ ์›์ฒœ์€ infra/ecr.tf์˜ ECR repository์ด๋ฉฐ, ์• ํ”Œ๋ฆฌ์ผ€์ด์…˜ ๋ฐฐํฌ๊ฐ€ ์™„๋ฃŒ๋˜์–ด์•ผ ํ…Œ์ŠคํŠธ๋ฅผ ์‹œ์ž‘ํ•  ์ˆ˜ ์žˆ๋‹ค.

์„œ๋น„์Šค ์ด๋ฏธ์ง€ ์›์ฒœ
Nginx moyeota/nginx (Amazon ECR)
Frontend moyeota/fe (Amazon ECR)
Backend moyeota/be (Amazon ECR)
AI/FastAPI moyeota/ai (Amazon ECR)

docker ps์— loadtest-node-exporter์™€ loadtest-cadvisor๋งŒ ๋ณด์ด๋ฉด ๊ด€์ธก ์ปจํ…Œ์ด๋„ˆ๋งŒ ์ค€๋น„๋œ ๊ฒƒ์ด๋ฉฐ, ๋ถ€ํ•˜ ๋Œ€์ƒ ์• ํ”Œ๋ฆฌ์ผ€์ด์…˜์€ ์•„์ง ์ค€๋น„๋˜์ง€ ์•Š์€ ์ƒํƒœ๋‹ค. ์• ํ”Œ๋ฆฌ์ผ€์ด์…˜์€ ECR digest, compose.yaml, runtime.env, release.env๋ฅผ ๊ธฐ์กด ๋ฐฐํฌ ์ ˆ์ฐจ(SSM/CI)๋กœ ๋ฐฐํฌํ•œ๋‹ค. loadtest-infra์˜ Prometheus/Grafana compose๊ฐ€ ์• ํ”Œ๋ฆฌ์ผ€์ด์…˜์„ ๋Œ€์‹  ์‹คํ–‰ํ•˜์ง€ ์•Š๋Š”๋‹ค.

3.3 ๋ชจ๋‹ˆํ„ฐ๋ง ์ปจํ…Œ์ด๋„ˆ ์›์ฒœ

๋ชจ๋‹ˆํ„ฐ๋ง EC2์˜ user data๊ฐ€ Docker์™€ Compose๋ฅผ ์„ค์น˜ํ•˜๊ณ  ๋‹ค์Œ ์ด๋ฏธ์ง€๋ฅผ pullํ•œ๋‹ค.

์ปจํ…Œ์ด๋„ˆ ์ด๋ฏธ์ง€ ์—ญํ• 
Prometheus prom/prometheus:v3.13.3 5์ดˆ scrape, ์‹œ๊ณ„์—ด ์ €์žฅ
Grafana grafana/grafana:13.2.1 ๋Œ€์‹œ๋ณด๋“œ
Node Exporter quay.io/prometheus/node-exporter:v1.12.1 V1 ํ˜ธ์ŠคํŠธ ์ง€ํ‘œ
cAdvisor ghcr.io/google/cadvisor:v0.60.5 ์ปจํ…Œ์ด๋„ˆ ์ง€ํ‘œ

Node Exporter์™€ cAdvisor๋Š” ๊ธฐ์กด V1 EC2์— SSM ๋ช…๋ น์œผ๋กœ ์„ค์น˜ํ•œ๋‹ค. Node Exporter๊ฐ€ host network๋กœ ์‹คํ–‰๋˜์–ด docker ps์— ํฌํŠธ๊ฐ€ ํ‘œ์‹œ๋˜์ง€ ์•Š๋Š” ๊ฒƒ์€ ์ •์ƒ์ด๋‹ค. cAdvisor๋Š” 8081:8080์œผ๋กœ ๋…ธ์ถœ๋œ๋‹ค.

3.4 k6 ์ฝ”๋“œ์™€ ์ž…๋ ฅ ํŒŒ์ผ

ํŒŒ์ผ ์šฉ๋„
smoke.js ๋‹ค์„ฏ ์ž‘์—…์˜ ๊ธฐ๋Šฅยท์ธ์ฆยท๊ฒฝ๋กœ ํ™•์ธ
http-mixed.js 4:4:3:1:1 ํ˜ผํ•ฉ HTTP RPS
http-operation.js ํ•˜๋‚˜์˜ ์ž‘์—… ๊ทธ๋ฃน๋งŒ ๊ณ ์ •ํ•ด T2ยทT4ยทT5ยทT6ยทT7 ์ธก์ •
event-burst.js ์‚ผ๊ฐํ˜• HTTP burst
hot-row.js ๋™์ผ ํ–‰๊ณผ ๋ถ„์‚ฐ ํ–‰์˜ ๋ณ‘๋ ฌ ์“ฐ๊ธฐ ๊ฒฝํ•ฉ
ws-connect.js ๋ฉ”์‹œ์ง€ ์—†๋Š” ๋™์‹œ WebSocket ์—ฐ๊ฒฐ
ws-message.js ๊ณ ์ • ์—ฐ๊ฒฐ ์ˆ˜์—์„œ aggregate ๋ฉ”์‹œ์ง€์œจ
lib/ ์ž…๋ ฅ ๊ฒ€์ฆ, ์ธ์ฆ, ํƒœ๊น…, ๊ณตํ†ต metric

์ž…๋ ฅ์€ test-data.example.json์„ ๋ณต์‚ฌํ•ด data/test-data.json์œผ๋กœ ๋งŒ๋“ ๋‹ค. ์‹ค์ œ ๊ฒฝ๋กœยทIDยทpayloadยทtest token์„ ์ž…๋ ฅํ•˜๊ณ , data/test-data.json์€ Git์— ์ปค๋ฐ‹ํ•˜์ง€ ์•Š๋Š”๋‹ค. ์„ ํƒํ•œ scenario์˜ ํ•„์ˆ˜ group์— REPLACE_๊ฐ€ ๋‚จ์•„ ์žˆ์œผ๋ฉด ์‹คํ–‰ ์ „์— ์ค‘๋‹จ๋œ๋‹ค.

cp loadtest-scenarios/data/test-data.example.json \
   loadtest-scenarios/data/test-data.json
mkdir -p loadtest-scenarios/results

์“ฐ๊ธฐ ์š”์ฒญ์€ ์ „์šฉ test account์™€ disposable/idempotent ๋ฐ์ดํ„ฐ๋ฅผ ์‚ฌ์šฉํ•œ๋‹ค. 409๋ฅผ ์„ฑ๊ณต์œผ๋กœ ๋‘˜ ๊ฒฝ์šฐ์—๋Š” ์˜ˆ์ƒ๋˜๋Š” ์ค‘๋ณตยท์ •์› ์ดˆ๊ณผ ๊ฒฐ๊ณผ์ธ์ง€ ๋ณ„๋„ ์ •ํ•ฉ์„ฑ ๊ฒ€์ฆ์„ ์ˆ˜ํ–‰ํ•ด์•ผ ํ•œ๋‹ค.

3.5 ์‹œ์ž‘ ๋ช…๋ น

๋จผ์ € public ๊ฒฝ๋กœ๋กœ smoke๋ฅผ ์‹คํ–‰ํ•œ๋‹ค.

k6 run \
  -e BASE_URL=https://PUBLIC_DOMAIN \
  -e DATA_FILE=./data/test-data.json \
  -e RUN_ID=smoke-01 \
  --summary-export=loadtest-scenarios/results/smoke-01.json \
  loadtest-scenarios/smoke.js

๊ธฐ์ค€ ํ˜ผํ•ฉ ๋ถ€ํ•˜๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™์ด ์‹คํ–‰ํ•œ๋‹ค.

k6 run \
  -e BASE_URL=https://PUBLIC_DOMAIN \
  -e DATA_FILE=./data/test-data.json \
  -e TARGET_RPS=50 \
  -e WARMUP=5m \
  -e DURATION=10m \
  -e RUN_ID=http-50-01 \
  --summary-export=loadtest-scenarios/results/http-50-01-summary.json \
  --out json=loadtest-scenarios/results/http-50-01-raw.json \
  loadtest-scenarios/http-mixed.js

์‹คํ–‰ ์ „ํ›„์— RUN_ID, git/image digest, ๋Œ€์ƒ public URL, ๋กœ์ปฌ ์žฅ๋น„, ์‹œ๊ฐ„๋Œ€, ๋™์‹œ WS ์ˆ˜, ํ…Œ์ŠคํŠธ data ๋ฒ„์ „์„ ๊ธฐ๋กํ•œ๋‹ค.

4. ๊ด€์ธก ๊ตฌ์กฐ์™€ ๊ฒ€์ฆ๊ฐ’

4.1 ์ˆ˜์ง‘ ๊ฒฝ๋กœ

๊ณ„์ธต ์ˆ˜์ง‘ ๋ฐฉ๋ฒ• ํ•ต์‹ฌ ๊ฐ’
k6 ๋กœ์ปฌ summary + JSON output ์‹ค์ œ RPS, dropped iterations, p50/p95/p99, status, checks, timeout
ํ˜ธ์ŠคํŠธ Node Exporter CPU, load, memory available, filesystem, disk I/O, network, FD
์ปจํ…Œ์ด๋„ˆ cAdvisor container CPU, working set, throttling, network, filesystem
Spring Micrometer /actuator/prometheus HTTP p95, Tomcat busy/max, JVM heap/direct, GC, Hikari active/pending
MySQL mysqld-exporter Threads, row lock wait/deadlock, buffer pool, redo, fsync, disk latency
Nginx exporter ๋˜๋Š” access log active/reading/writing/waiting, upstream time, 4xx/5xx, accept ์‹คํŒจ
WebSocket k6 custom + ์„œ๋ฒ„ ์ง€ํ‘œ open/close/error, handshake, ESTABLISHED/TIME_WAIT, message delivery

Prometheus ๊ธฐ๋ณธ scrape interval์€ 5์ดˆ๋‹ค. Prometheus Targets ํŽ˜์ด์ง€ ๋˜๋Š” ๋‹ค์Œ PromQL๋กœ exporter ์ƒํƒœ๋ฅผ ๋จผ์ € ํ™•์ธํ•œ๋‹ค.

up{job=~"v1-node|v1-cadvisor|v1-spring|v1-mysql|v1-nginx"}

๊ด€์ธก endpoint๊ฐ€ ์—†๋Š” ๊ณ„์ธต์€ ์›์ธ ๋ถ„์„์ด ์•„๋‹ˆ๋ผ ๊ฒฐ๊ณผ ํ™•์ธ๋งŒ ๊ฐ€๋Šฅํ•˜๋‹ค. ํŠนํžˆ SpringยทMySQLยทNginx exporter๋Š” ์ž๊ฒฉ ์ฆ๋ช…๊ณผ ์„œ๋น„์Šค ์„ค์ •์ด ํ•„์š”ํ•˜๋ฏ€๋กœ Terraform state์— ๋น„๋ฐ€๋ฒˆํ˜ธ๋ฅผ ๋„ฃ์ง€ ์•Š๊ณ  ๋ณ„๋„ ๊ตฌ์„ฑํ•œ๋‹ค.

4.2 ๊ณตํ†ต ์ค‘๋‹จ ์กฐ๊ฑด

๋‹ค์Œ ์ค‘ ํ•˜๋‚˜๋ฉด ํ•ด๋‹น ๋‹จ๊ณ„๋ฅผ ์ฆ‰์‹œ ์ค‘๋‹จํ•˜๊ณ  ์ƒํƒœ๋ฅผ ๋ณด์กดํ•œ๋‹ค.

  • 5xx ๋˜๋Š” timeout์ด 2๋ถ„ ์ด์ƒ 5% ์ดˆ๊ณผ
  • OOM, container restart, disk full
  • ์ขŒ์„ ์ดˆ๊ณผยท์ž”์—ฌ ์ขŒ์„ ์Œ์ˆ˜ยท์ค‘๋ณต ์Šน์ธยท๋ฉ”์‹œ์ง€ ์œ ์‹ค/์ค‘๋ณต/์ˆœ์„œ ์˜ค๋ฅ˜
  • k6 ๋ชฉํ‘œ ์ „๋‹ฌ๋ฅ  95% ๋ฏธ๋งŒ ๋˜๋Š” dropped iterations ๋ฐœ์ƒ
  • DB๊ฐ€ ๋ณต๊ตฌ ์ž‘์—…์„ ์š”๊ตฌํ•˜๋Š” ๋น„์ •์ƒ ์ƒํƒœ

5. ์„œ๋น„์Šค ๋ณ‘๋ชฉ ๊ตฌ๊ฐ„๋ณ„ ์‹œ๋‚˜๋ฆฌ์˜ค

๋ชจ๋“  ์‹œ๋‚˜๋ฆฌ์˜ค๋Š” ํ•œ ๋ฒˆ์— ํ•œ ์ถ•๋งŒ ๋ฐ”๊พผ๋‹ค. ๊ฐ ๋‹จ๊ณ„๋Š” ๊ธฐ๋ณธ์ ์œผ๋กœ 5๋ถ„ warm-up โ†’ 10๋ถ„ measure โ†’ 3ํšŒ ๋ฐ˜๋ณต์ด๋ฉฐ, ์ธก์ • ์ค‘์—๋Š” ๊ธฐ์กด V1 ์• ํ”Œ๋ฆฌ์ผ€์ด์…˜์˜ ์ด๋ฏธ์ง€ยทDB snapshotยทCPU creditยทexporter ์ƒํƒœ๋ฅผ ๊ณ ์ •ํ•œ๋‹ค.

T2. Spring ์š”์ฒญ ์ฒ˜๋ฆฌ์™€ JVM

๋ณ‘๋ชฉ ์ •์˜

Nginx์™€ DB๊ฐ€ ์—ฌ์œ ๋กœ์šด๋ฐ Spring์˜ Tomcat thread, JVM CPU/GC, heap/direct memory, JSON serialization ๋˜๋Š” container throttling์ด ๋จผ์ € ํฌํ™”๋˜๋Š” ํ๋ฆ„์ด๋‹ค.

k6 HTTPS โ†’ Nginx โ†’ Spring Controller โ†’ service/serialization โ†’ response

๋ถ€ํ•˜ ์„ค๊ณ„

  1. DBยท์™ธ๋ถ€ API๋ฅผ ์‚ฌ์šฉํ•˜์ง€ ์•Š๋Š” ๊ฐ€๋ฒผ์šด light group์œผ๋กœ Spring ์ฒ˜๋ฆฌ ๊ธฐ์ค€์„ ์„ ์ธก์ •ํ•œ๋‹ค.
  2. ๊ฐ™์€ ์กฐ๊ฑด์—์„œ ์‹ค์ œ list, detail, congestion, write, auth๋ฅผ ๊ฐ๊ฐ http-operation.js๋กœ ์ธก์ •ํ•œ๋‹ค.
  3. 40 โ†’ 46 โ†’ 53 RPS๋กœ ๋ฒค์น˜๋งˆํฌ ์žฌํ˜„ ํ›„ 25 โ†’ 50 โ†’ 75 โ†’ 100 โ†’ 150 โ†’ 200 โ†’ 300 โ†’ 400 โ†’ 512 RPS๋กœ ์šฉ๋Ÿ‰์„ ํƒ์ƒ‰ํ•œ๋‹ค.
  4. http_req_duration p95/p99์™€ Tomcat busy/max, JVM CPUยทheapยทdirect, allocation rate, GC pause, process CPU, context switch, cgroup throttling์„ ๊ฐ™์€ ์‹œ๊ฐ„์ถ•์—์„œ ๊ธฐ๋กํ•œ๋‹ค.

์˜ˆ์‹œ:

k6 run \
  -e BASE_URL=https://PUBLIC_DOMAIN \
  -e DATA_FILE=./data/test-data.json \
  -e REQUEST_GROUP=light \
  -e TARGET_RPS=50 \
  -e WARMUP=5m -e DURATION=10m \
  -e RUN_ID=t2-light-50-01 \
  loadtest-scenarios/http-operation.js

ํŒ์ •

DBยท์™ธ๋ถ€ ํ˜ธ์ถœ ์ง€ํ‘œ๊ฐ€ ์•ˆ์ •์ ์ธ๋ฐ Tomcat busy/max์™€ CPU๊ฐ€ ํ•จ๊ป˜ ํฌํ™”๋˜๋ฉด Spring thread/CPU/์ง๋ ฌํ™” ๊ฒฝ๋กœ๊ฐ€ 1์ฐจ ๋ณ‘๋ชฉ์ด๋‹ค. GC pause์™€ heap๋งŒ ์ฆ๊ฐ€ํ•˜๋ฉด ๊ฐ์ฒด ์ƒ์„ฑยท์‘๋‹ต ํฌ๊ธฐยทheap sizing์„, throttling์ด ์ฆ๊ฐ€ํ•˜๋ฉด ์ปจํ…Œ์ด๋„ˆ quota์™€ EC2 ํฌ๊ธฐ๋ฅผ ๋จผ์ € ๊ฒ€ํ† ํ•œ๋‹ค.

T4. ์ƒ์„ธ ์กฐํšŒ, N+1๊ณผ ์ง๋ ฌํ™”

๋ณ‘๋ชฉ ์ •์˜

์ƒ์„ธ ์กฐํšŒ ํ•œ ๊ฑด์ด ์ฐธ์—ฌ์žยท์ •์‚ฐยท์ฑ„ํŒ… ์ด๋ ฅ์„ ์—ฌ๋Ÿฌ query๋กœ ์ฝ๊ฑฐ๋‚˜ ํฐ JSON์œผ๋กœ ์ง๋ ฌํ™”ํ•˜๋ฉด์„œ DB query ์ˆ˜, response bytes, Spring CPU/GC๊ฐ€ ์ฆ๊ฐ€ํ•˜๋Š” ํ๋ฆ„์ด๋‹ค.

๋ถ€ํ•˜ ์„ค๊ณ„

requests.detail์— ๋‹ค์Œ fixture๋ฅผ ์—ฌ๋Ÿฌ entry๋กœ ์ค€๋น„ํ•˜๊ณ  REQUEST_GROUP=detail ๋กœ ๊ณ ์ •ํ•œ๋‹ค.

  • ์กด์žฌํ•˜๋Š” ํŒŸ: ์ฐธ์—ฌ์ž 0๋ช…ยท์†Œ์ˆ˜ยทํฐ ๋ชฉ๋ก
  • ์กด์žฌํ•˜์ง€ ์•Š๋Š” ํŒŸ: 404๊ฐ€ ์ •์ƒ์ธ์ง€ ๋ช…์‹œ
  • ์ž‘์€/์ค‘๊ฐ„/ํฐ ์ฑ„ํŒ…ยท์ •์‚ฐ ์ด๋ ฅ
  • ๋™์ผ ID ๋ฐ˜๋ณต๊ณผ ์—ฌ๋Ÿฌ ID ๋ถ„์‚ฐ

๊ฐ fixture๋ฅผ ๊ฐ™์€ RPS์—์„œ ๋ฐ˜๋ณตํ•˜๊ณ  captureResponseBody๋ฅผ ํ•„์š”ํ•œ ๊ทธ๋ฃน์—๋งŒ ์ผœ์„œ response_bytes์™€ JSON assertion์„ ์ˆ˜์ง‘ํ•œ๋‹ค.

k6 run \
  -e BASE_URL=https://PUBLIC_DOMAIN \
  -e DATA_FILE=./data/test-data.json \
  -e REQUEST_GROUP=detail \
  -e TARGET_RPS=50 -e WARMUP=5m -e DURATION=10m \
  -e RUN_ID=t4-detail-large-50-01 \
  loadtest-scenarios/http-operation.js

์ธก์ •๊ฐ’๊ณผ ํŒ์ •

  • HTTP p50/p95/p99, status๋ณ„ ์˜ค๋ฅ˜์œจ, response bytes
  • request๋‹น DB query count ๋ฐ SQL ํ•ฉ๊ณ„์‹œ๊ฐ„
  • Spring serialization ์‹œ๊ฐ„, CPU, allocation, GC
  • Hikari active/usage/acquire/pending

DB ์‹œ๊ฐ„๊ณผ query ์ˆ˜๊ฐ€ ํ•จ๊ป˜ ์ฆ๊ฐ€ํ•˜๋ฉด N+1ยท์ธ๋ฑ์Šคยทfetch ์ „๋žต์„ ์šฐ์„  ํ™•์ธํ•œ๋‹ค. DB ์‹œ๊ฐ„์€ ์•ˆ์ •์ ์ธ๋ฐ ์ „์ฒด ์‘๋‹ต๊ณผ response_bytes๋งŒ ์ฆ๊ฐ€ํ•˜๋ฉด ๊ฐ์ฒด ๊ทธ๋ž˜ํ”„์™€ JSON ์ง๋ ฌํ™”ยท์‘๋‹ต ํฌ๊ธฐ๊ฐ€ ๋ณ‘๋ชฉ์ด๋‹ค. ์—†๋Š” ID๋งŒ ๋А๋ฆฌ๋ฉด ์˜ˆ์™ธ ์ฒ˜๋ฆฌ๋‚˜ ๊ฒ€์ฆ query๋ฅผ ๋ณ„๋„๋กœ ํ™•์ธํ•œ๋‹ค.

T5. ํ˜ผ์žก๋„ ์™ธ๋ถ€ ํ˜ธ์ถœ

๋ณ‘๋ชฉ ์ •์˜

ํ˜ผ์žก๋„ ๊ณ„์‚ฐ ์š”์ฒญ์ด Spring์—์„œ FastAPI/์™ธ๋ถ€ LLM์„ ๋™๊ธฐ ํ˜ธ์ถœํ•˜๊ณ , ์™ธ๋ถ€ ์ง€์—ฐ์ด Tomcat threadยทoutbound poolยทHikari ๋˜๋Š” transaction ์ ์œ ๋กœ ์ „ํŒŒ๋˜๋Š” ํ๋ฆ„์ด๋‹ค.

HTTP โ†’ Spring โ†’ outbound connection pool โ†’ FastAPI/์™ธ๋ถ€ API โ†’ Spring response

๋ถ€ํ•˜ ์„ค๊ณ„

  1. requests.congestion์— ๋™์ผ spot๊ณผ 10ยท100๊ฐœ spot ๋ถ„์‚ฐ fixture๋ฅผ ์ค€๋น„ํ•œ๋‹ค.
  2. ์™ธ๋ถ€ ์˜์กด์„ฑ์˜ ํ…Œ์ŠคํŠธ ํ”„๋กœํŒŒ์ผ์„ ์• ํ”Œ๋ฆฌ์ผ€์ด์…˜ ์„ค์ • ๋˜๋Š” ๋ณ„๋„ stub์œผ๋กœ ๊ณ ์ •ํ•œ๋‹ค. ์ง€์—ฐ์€ 50ยท200ยท1,000ยท3,000ms, ์˜ค๋ฅ˜์œจ์€ 0ยท1ยท5ยท20%๋กœ ๋ถ„๋ฆฌํ•œ๋‹ค.
  3. ๊ฐ profile์—์„œ http-operation.js์˜ REQUEST_GROUP=congestion์„ ๋™์ผ RPS๋กœ ์‹คํ–‰ํ•œ๋‹ค. ์ง€์—ฐยท์˜ค๋ฅ˜๋ฅผ k6๊ฐ€ ์ž„์˜๋กœ ์ฃผ์ž…ํ•˜๋Š” ๊ฒƒ์ด ์•„๋‹ˆ๋ฏ€๋กœ, ํ•ด๋‹น profile์„ ํ†ต์ œํ•  ์ˆ˜ ์—†์œผ๋ฉด ์›์ธ ๋ถ„๋ฆฌ ํ…Œ์ŠคํŠธ๋กœ ๊ฐ„์ฃผํ•˜์ง€ ์•Š๋Š”๋‹ค.
  4. retry๊ฐ€ ํ™œ์„ฑํ™”๋˜์–ด ์žˆ์œผ๋ฉด retry ํšŸ์ˆ˜์™€ backoff๋ฅผ ๊ณ ์ •ํ•˜๊ณ  ๋ณ„๋„ ๊ฒฐ๊ณผ๋กœ ๊ธฐ๋กํ•œ๋‹ค.

์ธก์ •๊ฐ’๊ณผ ํŒ์ •

  • Spring HTTP p95/p99, Tomcat busy/max, request queue
  • outbound active/pending, timeout, retry, FastAPI worker queue
  • Hikari active/pending ๋ฐ transaction duration
  • ์™ธ๋ถ€ ์˜ค๋ฅ˜์œจ๊ณผ ์‚ฌ์šฉ์ž ์‘๋‹ต ์˜ค๋ฅ˜์œจ์˜ ์ „ํŒŒ ๋น„์œจ

์™ธ๋ถ€ ์ง€์—ฐ์— Tomcatยทoutbound pending์ด ๋น„๋ก€ํ•˜๊ณ  DB๋„ ํ•จ๊ป˜ ์ ์œ ๋˜๋ฉด ๋™๊ธฐ ํ˜ธ์ถœ ๋˜๋Š” transaction ๋ฒ”์œ„๊ฐ€ ๋„“์€ ๊ฒƒ์ด๋‹ค. ์™ธ๋ถ€ ์˜ค๋ฅ˜๋ณด๋‹ค ์‚ฌ์šฉ์ž ์˜ค๋ฅ˜๊ฐ€ ํฌ๋ฉด retry stormยทfallbackยทtimeout ์ฒ˜๋ฆฌ๋ฅผ ๋ณ„๋„๋กœ ํŒ๋‹จํ•œ๋‹ค.

T6. ํ˜ผ์žก๋„ ์ตœ์‹ ๊ฐ’ upsert์™€ ์ด๋ ฅ ์ €์žฅ

๋ณ‘๋ชฉ ์ •์˜

์™ธ๋ถ€ ๊ณ„์‚ฐ ๊ฒฐ๊ณผ๋ฅผ ์ตœ์‹ ๊ฐ’ row์— upsertํ•˜๊ฑฐ๋‚˜ ์ด๋ ฅ row๋กœ insertํ•  ๋•Œ ๋™์ผ spot์˜ row lock, redo/fsync, ์ธ๋ฑ์Šค์™€ Hikari ๋Œ€๊ธฐ๊ฐ€ ์ฆ๊ฐ€ํ•˜๋Š” ํ๋ฆ„์ด๋‹ค.

๋ถ€ํ•˜ ์„ค๊ณ„

์™ธ๋ถ€ ๊ฒฐ๊ณผ๋ฅผ ๊ณ ์ •ํ•œ ํ…Œ์ŠคํŠธ profile์—์„œ ๋‹ค์Œ ๋‘ endpoint group์„ ๋ณ„๋„๋กœ ๋งŒ๋“ ๋‹ค.

  • congestionWriteLatest: ์ตœ์‹ ๊ฐ’ upsert๋งŒ ์ˆ˜ํ–‰
  • congestionWriteHistory: ์ตœ์‹ ๊ฐ’ upsert + ์ด๋ ฅ insert ์ˆ˜ํ–‰

๊ฐ group์— ๋™์ผ spot, 10๊ฐœ spot, 100๊ฐœ spot fixture๋ฅผ ์ค€๋น„ํ•˜๊ณ  http-operation.js๋ฅผ 1 โ†’ 5 โ†’ 10 โ†’ 20 โ†’ 40 โ†’ 80 writes/s๋กœ ์‹คํ–‰ํ•œ๋‹ค. ์ตœ์‹ ๊ฐ’๊ณผ ์ด๋ ฅ ์ €์žฅ์„ ๋™์‹œ์— ์ผœ์ง€ ์•Š๊ณ  ๊ฐ๊ฐ ์ธก์ •ํ•ด์•ผ ์™ธ๋ถ€ ํ˜ธ์ถœ ์˜ํ–ฅ์„ ์ œ๊ฑฐํ•  ์ˆ˜ ์žˆ๋‹ค.

์ธก์ •๊ฐ’๊ณผ ํŒ์ •

  • Innodb_row_lock_time, current waits, deadlock
  • Threads_running, MySQL CPU, buffer pool hit
  • redo bytes, fsync, dirty page, EBS queue/await
  • Hikari pending, transaction duration, write p95/p99

๋™์ผ spot์—์„œ๋งŒ ๊ธ‰๊ฒฉํžˆ ๋‚˜๋น ์ง€๋ฉด hot-row lock, ๋ชจ๋“  spot์—์„œ ๋‚˜๋น ์ง€๋ฉด redoยทI/Oยท์ธ๋ฑ์Šค ๋น„์šฉ์œผ๋กœ ๋ถ„๋ฅ˜ํ•œ๋‹ค. deadlockยท์ค‘๋ณต ์ด๋ ฅยท์ตœ์‹ ๊ฐ’ ์œ ์‹ค์€ latency์™€ ๋ฌด๊ด€ํ•˜๊ฒŒ ์‹คํŒจ๋‹ค.

T7. ์ฐธ์—ฌ ์š”์ฒญ ์ €์žฅ ๊ธฐ์ค€์„ 

๋ณ‘๋ชฉ ์ •์˜

์ขŒ์„ ์ •์› ๊ฒฝํ•ฉ์„ ์ œ์™ธํ•˜๊ณ  ์„œ๋กœ ๋‹ค๋ฅธ ํŒŸ์— ์ฐธ์—ฌ ์š”์ฒญ์„ ๋ถ„์‚ฐํ–ˆ์„ ๋•Œ์˜ ์ˆœ์ˆ˜ insert ์ฒ˜๋ฆฌ๋Ÿ‰์„ ๊ตฌํ•˜๋Š” ๋‹จ๊ณ„๋‹ค.

๋ถ€ํ•˜ ์„ค๊ณ„

requests.write์— ์„œ๋กœ ๋‹ค๋ฅธ ํŒŸ ID์™€ ์‚ฌ์šฉ์žยท๋ฉฑ๋“ฑ request ID๋ฅผ ์ถฉ๋ถ„ํžˆ ์ค€๋น„ํ•œ๋‹ค. http-operation.js์—์„œ ์“ฐ๊ธฐ group๋งŒ ๊ณ ์ •ํ•˜๊ณ  1 โ†’ 5 โ†’ 10 โ†’ 20 โ†’ 40 โ†’ 80 writes/s๋ฅผ ์‹คํ–‰ํ•œ๋‹ค.

k6 run \
  -e BASE_URL=https://PUBLIC_DOMAIN \
  -e DATA_FILE=./data/test-data.json \
  -e REQUEST_GROUP=write \
  -e TARGET_RPS=20 -e WARMUP=5m -e DURATION=10m \
  -e RUN_ID=t7-write-20-01 \
  loadtest-scenarios/http-operation.js

์ธก์ •๊ฐ’๊ณผ ํŒ์ •

์„ฑ๊ณตยท์ค‘๋ณตยท์ œ์•ฝ์กฐ๊ฑด ์˜ค๋ฅ˜๋ฅผ status์™€ business result๋กœ ๋ถ„๋ฆฌํ•˜๊ณ  insert p95/p99, Hikari pending, MySQL CPU/lock/I/O, commitยทredo๋ฅผ ๊ธฐ๋กํ•œ๋‹ค. ๋ถ„์‚ฐ insert๊ฐ€ ์•ˆ์ •์ ์ธ๋ฐ ๋™์ผ ํŒŸ ์Šน์ธ๋งŒ ์•…ํ™”๋˜๋ฉด ์ผ๋ฐ˜ ์“ฐ๊ธฐ ์ฒ˜๋ฆฌ๋Ÿ‰๊ณผ ์ขŒ์„ hot-row ๊ฒฝํ•ฉ์„ ํ˜ผ๋™ํ•˜์ง€ ์•Š๋Š”๋‹ค.

T8. ์ธ๊ธฐ ํŒŸ ์Šน์ธ hot-row

๋ณ‘๋ชฉ ์ •์˜

๋™์ผ ํŒŸ์˜ ์ •์›/์Šน์ธ row๋ฅผ ์—ฌ๋Ÿฌ ์š”์ฒญ์ด ๋™์‹œ์— ๊ฐฑ์‹ ํ•˜๋ฉด์„œ DB row lock, deadlock, Hikari ๋Œ€๊ธฐ์™€ ์ •ํ•ฉ์„ฑ ์œ„๋ฐ˜์ด ๋ฐœ์ƒํ•˜๋Š” ํ๋ฆ„์ด๋‹ค.

๋ถ€ํ•˜ ์„ค๊ณ„

hot-row.js๊ฐ€ ํ•œ round๋งˆ๋‹ค http.batch๋กœ ๋™์ผ ์‹œ์ ์— ์š”์ฒญ์„ ๋ฐœํ–‰ํ•˜๊ณ , round ๋’ค hotRowVerify๋ฅผ ํ˜ธ์ถœํ•œ๋‹ค.

  • ๋™์ผ ํŒŸ: REQUEST_GROUP=hotRowSame
  • ๋Œ€์กฐ๊ตฐ: REQUEST_GROUP=hotRowDistributed
  • ๋™์‹œ์„ฑ: CONCURRENCY=2,5,10,20,50,100
  • ๊ธฐ๋ณธ round: 3ํšŒ, round ๊ฐ„๊ฒฉ 10์ดˆ
  • ์˜ˆ์ƒ status๋Š” ์‹ค์ œ ๋น„์ฆˆ๋‹ˆ์Šค ๊ฒฐ๊ณผ๋งŒ ํ—ˆ์šฉํ•˜๊ณ , 409๋ฅผ ํ—ˆ์šฉํ•˜๋ฉด ์ค‘๋ณต/์ •์› ์ดˆ๊ณผ ๊ฒฐ๊ณผ์ž„์„ ๋ณ„๋„ ๊ธฐ๋ก
k6 run \
  -e BASE_URL=https://PUBLIC_DOMAIN \
  -e DATA_FILE=./data/test-data.json \
  -e REQUEST_GROUP=hotRowSame \
  -e VERIFY_GROUP=hotRowVerify \
  -e CONCURRENCY=20 -e ROUNDS=3 -e ROUND_PAUSE=10s \
  -e RUN_ID=t8-same-20-01 \
  loadtest-scenarios/hot-row.js

hotRowVerify.assertions์˜ JSON path์™€ ์ •์› ๊ฐ’์€ ์‹ค์ œ API ์‘๋‹ต์— ๋งž๊ฒŒ ์ˆ˜์ •ํ•œ๋‹ค. ์ž”์—ฌ ์ขŒ์„ ์Œ์ˆ˜, ์Šน์ธ ์ˆ˜ ์ดˆ๊ณผ, ์ค‘๋ณต ์Šน์ธ, deadlock์€ ๋‹จ ํ•œ ๊ฑด์ด์–ด๋„ ์‹คํŒจ๋‹ค. ๋™์ผ ํ–‰๊ณผ ๋ถ„์‚ฐ ํ–‰์˜ latency ์ฐจ์ด๊ฐ€ row lock์˜ ์ฆ๊ฑฐ๊ฐ€ ๋œ๋‹ค.

T9. WebSocket ์—ฐ๊ฒฐ ์ž์ฒด

๋ณ‘๋ชฉ ์ •์˜

๋ฉ”์‹œ์ง€๋ฅผ ๋ณด๋‚ด์ง€ ์•Š๊ณ  WebSocket handshake, Nginx proxy, Spring session, file descriptor, heap/direct memory๊ฐ€ ๋™์‹œ ์—ฐ๊ฒฐ ์ˆ˜์— ๋”ฐ๋ผ ํฌํ™”๋˜๋Š” ํ๋ฆ„์ด๋‹ค.

๋ถ€ํ•˜ ์„ค๊ณ„

ws-connect.js๋ฅผ local process์—์„œ ์‹คํ–‰ํ•œ๋‹ค.

  • ์ˆ˜์š” ์žฌํ˜„: 1,575 โ†’ 1,800 โ†’ 2,100 connections
  • ์šฉ๋Ÿ‰ ํƒ์ƒ‰: 500 โ†’ 1,000 โ†’ 1,500 โ†’ 3,000 โ†’ 5,000 connections
  • ์—ฐ๊ฒฐ๋ฅ : 5.25 โ†’ 6 โ†’ 7 โ†’ 10 โ†’ 20 โ†’ 50 connections/s
  • ๊ฐ ๋‹จ๊ณ„ idle hold 30๋ถ„, ramp-down 1๋ถ„
k6 run \
  -e WS_URL=wss://PUBLIC_DOMAIN \
  -e DATA_FILE=./data/test-data.json \
  -e TARGET_CONNECTIONS=1800 \
  -e CONNECT_RATE=6 \
  -e HOLD_DURATION=30m \
  -e RUN_ID=t9-ws-1800-01 \
  loadtest-scenarios/ws-connect.js

์ธก์ •๊ฐ’๊ณผ ํŒ์ •

handshake p95/p99ยท์„ฑ๊ณต๋ฅ , k6์˜ open/close/error, Nginx active/reading/writing/ waiting, Spring FDยทESTABLISHED/TIME_WAIT, heap/direct memoryยทGCยทheartbeat, host/container CPU๋ฅผ ๊ธฐ๋กํ•œ๋‹ค. ๋กœ์ปฌ file descriptorยทCPUยท๋„คํŠธ์›Œํฌ๊ฐ€ ๋จผ์ € ํฌํ™”๋˜๋ฉด ์„œ๋ฒ„ ํ•œ๊ณ„๋กœ ํŒ์ •ํ•˜์ง€ ๋ง๊ณ  generator headroom ์‹คํŒจ๋กœ ํ‘œ์‹œํ•œ๋‹ค.

T10. WebSocket ๋ฉ”์‹œ์ง€ยทfan-outยท์ €์žฅ

๋ณ‘๋ชฉ ์ •์˜

๊ณ ์ •๋œ WebSocket ์—ฐ๊ฒฐ์—์„œ inbound message๊ฐ€ Spring fan-out, outbound delivery, DB commit๊ณผ ๊ฒฐํ•ฉ๋˜์–ด REST์™€ ๊ณต์œ  ์ž์›์„ ์••๋ฐ•ํ•˜๋Š” ํ๋ฆ„์ด๋‹ค.

๋ถ€ํ•˜ ์„ค๊ณ„

ws-message.js์—์„œ ์—ฐ๊ฒฐ ์ˆ˜๋ฅผ ๊ณ ์ •ํ•˜๊ณ  aggregate MESSAGE_RATE๋ฅผ ๋ณ€๊ฒฝํ•œ๋‹ค.

  • ์˜ˆ์ƒ inbound: 31.5 โ†’ 36 โ†’ 42 msg/s
  • ์šฉ๋Ÿ‰ ํƒ์ƒ‰: 10 โ†’ 20 โ†’ 50 โ†’ 100 โ†’ 300 msg/s
  • ์—ฐ๊ฒฐ ์ˆ˜: 1,575ยท1,800ยท2,100 ๋ฐ ํ•„์š” ์‹œ 3,000
  • ๋ฐฉ ๋ถ„ํฌ: ํ•œ ๋ฐฉ ์ง‘์ค‘, ์—ฌ๋Ÿฌ ๋ฐฉ ๋ถ„์‚ฐ
  • message profile: ์ €์žฅ ์—†๋Š” heartbeat, DB ์ €์žฅ chat message
  • fan-out: ํ‰๊ท  ์ˆ˜์‹ ์ž 1ยท3ยท9๋ช…
k6 run \
  -e WS_URL=wss://PUBLIC_DOMAIN \
  -e DATA_FILE=./data/test-data.json \
  -e TARGET_CONNECTIONS=1800 \
  -e MESSAGE_RATE=36 \
  -e CONNECTION_RAMP=5m -e DURATION=10m \
  -e RUN_ID=t10-message-36-01 \
  loadtest-scenarios/ws-message.js

๋ฉ”์‹œ์ง€ payload์—๋Š” runId, VU, iteration, timestamp๋ฅผ ํฌํ•จํ•ด ์ˆ˜์‹  ์œ ์‹คยท์ค‘๋ณตยท ์ˆœ์„œ ์˜ค๋ฅ˜๋ฅผ ์‹๋ณ„ํ•œ๋‹ค. end-to-end delivery p95/p99, sent/received/lost/duplicate, outbound deliveries/s, Spring CPU, network, DB commit, Hikari pending์„ ํ•จ๊ป˜ ๊ธฐ๋กํ•œ๋‹ค. ๋ฉ”์‹œ์ง€ ์ €์žฅ ์‹œ์—๋งŒ REST p95๊ฐ€ ์•…ํ™”๋˜๋ฉด ์ €์žฅ ๊ฒฝ๋กœ ๋น„๋™๊ธฐํ™” ๋˜๋Š” DB write path ๋ถ„๋ฆฌ๋ฅผ ์šฐ์„  ๊ฒ€ํ† ํ•œ๋‹ค.

T11. RESTยทWebSocket ๊ณต์œ  ์ž์› ๊ฐ„์„ญ

๋ณ‘๋ชฉ ์ •์˜

WebSocket connection/session/fan-out์ด REST์™€ ๊ฐ™์€ Spring process, heap, thread, Hikari, host network๋ฅผ ์‚ฌ์šฉํ•ด HTTP p95 ๋˜๋Š” ์˜ค๋ฅ˜์œจ์„ ์•…ํ™”์‹œํ‚ค๋Š” ํ๋ฆ„์ด๋‹ค.

๋ถ€ํ•˜ ์„ค๊ณ„

  1. WebSocket ์—†์ด http-mixed.js๋ฅผ TARGET_RPS=50์œผ๋กœ ์‹คํ–‰ํ•ด control์„ ๋งŒ๋“ ๋‹ค.
  2. ๋ณ„๋„ ๋กœ์ปฌ process์—์„œ ws-connect.js๋ฅผ ์‹œ์ž‘ํ•˜๊ณ  hold ๋‹จ๊ณ„์— ์ง„์ž…์‹œํ‚จ๋‹ค.
  3. WS ์—ฐ๊ฒฐ์„ 1,575ยท1,800ยท2,100ยท3,000ยท5,000์œผ๋กœ ๊ณ ์ •ํ•œ ์ƒํƒœ์—์„œ ๋™์ผํ•œ HTTP ํ˜ผํ•ฉ ๋ถ€ํ•˜๋ฅผ ๋ฐ˜๋ณตํ•œ๋‹ค.
  4. ํ•„์š”ํ•˜๋ฉด T10์˜ 36 msg/s๋ฅผ ์ถ”๊ฐ€ํ•ด ์—ฐ๊ฒฐ๋งŒ์˜ ๊ฐ„์„ญ๊ณผ ๋ฉ”์‹œ์ง€ fan-out ๊ฐ„์„ญ์„ ๋ถ„๋ฆฌํ•œ๋‹ค.
# ํ„ฐ๋ฏธ๋„ 1: WS hold
k6 run -e WS_URL=wss://PUBLIC_DOMAIN \
  -e DATA_FILE=./data/test-data.json \
  -e TARGET_CONNECTIONS=1800 -e CONNECT_RATE=6 -e HOLD_DURATION=30m \
  -e RUN_ID=t11-ws-1800-01 loadtest-scenarios/ws-connect.js

# ํ„ฐ๋ฏธ๋„ 2: ๊ฐ™์€ ์‹œ๊ฐ„๋Œ€ REST
k6 run -e BASE_URL=https://PUBLIC_DOMAIN \
  -e DATA_FILE=./data/test-data.json \
  -e TARGET_RPS=50 -e WARMUP=5m -e DURATION=10m \
  -e RUN_ID=t11-http-with-ws-1800-01 loadtest-scenarios/http-mixed.js

ํŒ์ •

WS๊ฐ€ ์—†๋Š” control ๋Œ€๋น„ REST p95๊ฐ€ 20% ์ด์ƒ ์•…ํ™”๋˜๊ณ  Spring CPU/heap/GC, Tomcat busy, Hikari ๋˜๋Š” network ์ƒ๊ด€๊ด€๊ณ„๊ฐ€ ํ™•์ธ๋˜๋ฉด ๊ณต์œ  process/resource ๋ถ„๋ฆฌ๊ฐ€ ์šฐ์„ ์ด๋‹ค. ๋กœ์ปฌ generator ์ง€ํ‘œ๊ฐ€ ๋จผ์ € ํฌํ™”๋œ ๊ฒฝ์šฐ์—๋Š” ๊ฒฐ๊ณผ๋ฅผ ์„œ๋ฒ„ ํ•œ๊ณ„๋กœ ์‚ฌ์šฉํ•˜์ง€ ์•Š๋Š”๋‹ค.

6. ๊ฒฐ๊ณผ๋ฌผ๊ณผ ํ™•์žฅ ๋ฐฉํ–ฅ ๊ฒฐ์ •

๊ฐ ์‹คํ–‰์€ ๋‹ค์Œ์„ ๋‚จ๊ธด๋‹ค.

  • scenario, RUN_ID, ๋ชฉํ‘œ/์‹ค์ œ RPS ๋˜๋Š” WS ์—ฐ๊ฒฐยท๋ฉ”์‹œ์ง€์œจ
  • p50/p95/p99, status๋ณ„ ์˜ค๋ฅ˜์œจ, checks, timeout, dropped iterations
  • ์ตœ์ดˆ SLO ์œ„๋ฐ˜ ์‹œ๊ฐ๊ณผ C_knee, C_safe
  • ํ˜ธ์ŠคํŠธยท์ปจํ…Œ์ด๋„ˆยทSpringยทMySQLยทNginx ์ง€ํ‘œ์˜ ๋™์ผ ์‹œ๊ฐ„ ๋ฒ”์œ„
  • ์ขŒ์„ยท๋ฉ”์‹œ์ง€ ์ •ํ•ฉ์„ฑ ๊ฒฐ๊ณผ
  • ๋กœ์ปฌ generator CPU/memory/FD/network headroom
  • k6 summary/raw JSON, Grafana screenshot, image digest์™€ DB snapshot ์ •๋ณด

ํŒ์ • ์˜ˆ์‹œ๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™๋‹ค.

๊ด€์ธก ์ฆ๊ฑฐ ์šฐ์„  ๊ฒ€ํ† ํ•  ํ™•์žฅ ๋ฐฉํ–ฅ
Host CPUยทcontainer throttling์ด ๋จผ์ € ํฌํ™”, DB ์—ฌ์œ  EC2 ํฌ๊ธฐ์™€ container quota ์žฌ๋ฐฐ๋ถ„
MySQL CPUยทlockยทI/O์™€ Hikari pending์ด ํ•จ๊ป˜ ์ฆ๊ฐ€ query/transaction ๊ฐœ์„  ํ›„ DB ๋ถ„๋ฆฌ ๋˜๋Š” RDS
์™ธ๋ถ€ API ์ง€์—ฐ์— Tomcatยทoutbound pending์ด ๋น„๋ก€ timeout, cache, single-flight, bulkhead, ๋น„๋™๊ธฐํ™”
WebSocket ์—ฐ๊ฒฐ๋งŒ์œผ๋กœ heapยทFD๊ฐ€ ํฌํ™” WebSocket gateway ๋˜๋Š” process ๋ถ„๋ฆฌ
๋ฉ”์‹œ์ง€ ์ €์žฅ ์‹œ์—๋งŒ REST๊ฐ€ ์•…ํ™” ์ฑ„ํŒ… ์ €์žฅ ๋น„๋™๊ธฐํ™” ๋˜๋Š” DB write path ๋ถ„๋ฆฌ
๋™์ผ row์—์„œ๋งŒ ์ง€์—ฐ ๊ธ‰์ฆยทdeadlock Redis ์›์ž ์นด์šดํ„ฐ/๋ฉฑ๋“ฑํ‚ค ๋“ฑ hot-row ๊ตฌ์กฐ ๋ณ€๊ฒฝ
C_safe < ๊ณ„ํš ํ”ผํฌ ร— ์šด์˜ ์—ฌ์œ ์œจ ์ตœ์ ํ™” ๋˜๋Š” ์ˆ˜์ง/์ˆ˜ํ‰ ํ™•์žฅ ํ›„ ๋™์ผ ์กฐ๊ฑด ์žฌ์‹œํ—˜

์ตœ์ข… ๋ณด๊ณ ์„œ๋Š” ์‹œ๋‚˜๋ฆฌ์˜ค๋ณ„ 3ํšŒ ๋ฐ˜๋ณต์˜ ์ค‘์•™๊ฐ’๊ณผ ๋ณ€๋™ํญ์„ ํ•จ๊ป˜ ์ œ์‹œํ•˜๊ณ , ๋ถ€ํ•˜ ๋ฐœ์ƒ๊ธฐ ํ•œ๊ณ„๋กœ ์ธํ•ด ๋ชฉํ‘œ๋ฅผ ๋‹ฌ์„ฑํ•˜์ง€ ๋ชปํ•œ ์‹คํ–‰์€ ์„œ๋ฒ„ ์šฉ๋Ÿ‰ ๊ณ„์‚ฐ์—์„œ ์ œ์™ธํ•œ๋‹ค.