Vortrag VII Stichpunkte - Archi-Lab/fae-team-3 GitHub Wiki
Vortrag Resilience
Resilience, Monitoring, Logging and disaster recovery
Komal Ahluwalia; Marion Bruns
concept of resilience
what is not: impervious to failure - not possible fighting against failure is not efficient, you cant rule out all possibliteis
avoid changes - changes are the biggest thread to software
resilience is meant to keep time between failures long, to recover quickly from failures
resilience: there is a failure --- but the system can recover from it
failures are maybe to expensive to avoid - tolerate failures
What is failure: simple: service down
complex: maybe high latency call to downstream takes ages => customers give up/timeout => service is practialy unavailable => failure
network failures database down or overloaded legacy systems unavailable service to service communication fails due to changes hdd broken failures due to design
Availabiltiy is not guranteed
known unkowns - database down, network
unknown unkowns - things you did not think about or forget
To ensure avalibility - first class citizens
continous delivery pipeline
monitoring and logging and tracking
tests!: unit tests, integration tests, consumer driven contract tests, functional tests, performance tests
everything as code! including infrastructure or CD pipelines
Handle failures gracefully:
scaling of services - heavy load
is it possible for the service?
concurrent database access
concurrent file access
concurrent storage access
autoscaling support in the cloud
avalibilty zones/multiple data center
scaling comes on costs: consistency, balance where the customer/business value is
feature toggles -
support trunk based development and continouous derlivery - release toggles
experiment toggle - try out new
permission toggle - specific tunctionality for specific users
can be applied on runtime, request deployment
decoupling toggle pint and the logic introduce router
rollout - a/b testing, canary release
configuration in static file, db, configuration file
adds testing complexity
circuit breaker -
decorate functional calls with a circuit braker - wrap function with circuit breaker
monitors call to maks
three states - closed, open, half open
define threshold before open, define time intervall for retry, define fallback
circuit breaker open - normal
closed - error
half open - try if things back to normal
costs: be carfeul when defining threshold, retry time -> overreaction
isolation - decoupling of services
make yourself independent of slow and failure prone services
- requestiong the service to deliver response in the future
event driven approach
loose coupling
caching
Be prepared:
run backups frequently
and test backups
shutdown service from time to timeout
chaos monkey
run a pre production environment
monitoring and logging
load testing
automatize as much as possibe
ifrastruture code
running rollacks
versioning of application docker images
Observability
you need to understand what is happening in your application
design an application in a way that they emit all relevant information
errors count when it comes to service avalability -
logging important
basic monitor is health check - define metrics: latency, durability, availability for each endpoint
check not only if the service is running, but check also if needed systems are working
Alerting!
reasonable conditions and thrsholds
you have to be alerted, if their are problems
to much can lead to ignorance
Desaster recovery
what do we do to recover from failures
mean time to recover
being informend - monitoring & alerting
dashboard - make failure visible - amount of time to fulfill request
taking action
prepare for when it fails
rollback
one click rollback - entire build and deployment flow needs to be automated
EVERYTHING - otherwise things will be forgotten
blue-green deployments
forward fix
if you know what the problem is
a small fix can fix everything - seldom
escalate
last possibility
clear plan - point of contacts, clear responsibilities
Orchestration
minimum: copy the current manual cinfiguration and put int into vcs
better: configure using pipeline as code plugins and vc the configuration files
biggest stress is change
failures are inevitable (especially in distribted environments)
fail as graceful as possible
setup a continouous delivery pipeline
Implement tests, starting from unit tests, integration tests, load tests
"i dont even know" - nah -- implement monitoring and logging at a very early stage use common librarys for logging, naming conventions, format conventions
define the metrics - what is important
automatisie as much as possible
stuff happens - deal with it
q/a:
implement only what you need
chaching is not that helpful most of the time
ai for monitoring or predicting failures not currently used - but in the not far future maybe possible