Vortrag 7 Notizen - Archi-Lab/fae-team-3 GitHub Wiki

resilience monitoring logging disaster recovery

what is resilience?

fighting all failures is not so good, unproductive

possible problems:

• service is down
• high load causing latency
• consumers of services don't wait or time out
• network failures
• hard disk dies
• a change in communication style

you can't know everything

CI Pipeline can make it easier, restarting services and testing it, automation

Scaling:

can it even scale:

• concurrent database access or file or storage

are there many data centres

ask:what provides business value.

Feature toggle overwiev

toggle release features, so it is easier to deactivate them. to quickly react to problems

keep the system resilient

toggles can be applied fast and during runtime or deployment or upon request

but it also adds complexity

circuit breaker:

timeout fallback

in case some connection fails. stopping connection tries. using fallback. later retry. open close half open

isolation:

queue asynchron

test your backups, they might not work, also use a pre produktion environment

automate as much as possible

observability

logging: clear format

see the errors

have a team dashboard visualiseit

save to db first

monitoring systems:

health check endpoints: but with useful info not just is alive but also are subsystems working

metrics

system metriks are important

push data to agent then to monitoring system and db

instrumentation:

alerting: define treshold send messages and such. give alert to people that something bad is going on but priorities.

disaster recovery:

meantime to rcover

monitoring alerting

have a dashboard with useful information

taking actions:

rollback: best with one click also blue green strategy

fix forward: can take a long time and no guarantee

escalate: inform people have someone set up to be informed

rchestration

failure is inevitable monitor and do logging automate have metrics

have a good execution environment