lessons_from_failures - chef-boneyard/chef-summit-2014 GitHub Wiki

Hindsight Bias - that was easy in hindsight Countual factuals - the fact is you didn’t

⁃	Don’t let zeal and bias for a solution (or hatred for

The tool is not usually the problem, its the process around the tool

  • everyone has their favourite tool, just accept and get on with it

Massive roll-out - fuckup at scale

  • roll-back bad changes
  • we can’t go back from using
  • common to not know the root password

Like guard, when you put the template on

chef template verification step - what it should do to verify - refuses to install an invalid template

roll out changes that break a subsystem vs. roll out a change that breaks the roll out system. (catastrophic change)

propagating changes to ambiguous

out of band system , e.g. enabling root console or rundeck in

collective with chef

package managers - can end up installing a later version than you wanted

  • can use your own repo (to block the install of a later version)
  • yum particularly vulnerable to this

upgrades available - can cause a

  • especially if available upstream
  • might not be you that made the change -

private repos - go out of date

  • packages you care about - explicitly pin the version you use in the package resource

unintended consequence

spacewalk - open source satalitte = mirror upstream repositories. pulp does a similar thing. allow things into dev but not production

cookbook broken dependency tree - chef can end up randomly choosing a

  • berkshelf

multi-cloud deployments with chef

  • startup move between clouds when the price is right
  • fundamental base platforms differ between providers
  • performance can really differ - means you can’t assume arbitrage
  • things out of your control really need to be trusted

CI/CD security, in

  • how do people version across
  • services sharing a common library
  • every change to the common library can cause a cascade of CI builds
  • most people don’t have infinite compute resources
  • don’t want a huge project / matrix
  • versioning applications , network configuration, package definiiotns
  • new software version, but no functional change, e.g. read me
  • having a way to present to the user a dashboard that customers can see (e.g. breaking, functional changes)

Q: when people first implement, what pain points do you see

  • we rebuilt the cookbooks, settling on different patterns
  • most of a team had not enough experience - gettg to grips with possibilities
  • finding and solving
  • learning curves 1) cook books 2) workflow and tooling
  • struggling a lot initially - its a lot to learn - training material give a leg up
  • docs were pretty terse - implications not clearly documented - a lot better id
  • using Ruby - getting people to contribute - make it easy e.g. to run a shell script at the end
  • intro chef: 1) its declarative language can be learnt top down 2) by the way its Ruby
  • a good chef recipe looks
  • execute execute execute, then show resources, documentation
  • people are comfortable not benign deep into man
  • lwrps look like recipes - so everyone - not a separate thing as in puppet
  • so much tooling - keeps changing -
  • newbies struggle with knowing which way to do things - e.g. roles, environment cookbooks
  • chef rfc - making future changes more transparency of what’s coming up, mitigates things breaking - and now you can get involved.
  • supporting monolithic repos plus one-repo-per-cookbooks
  • different situations like tightly regulated environments drive different assumptions and workflows (e.g. devs ever getting root access to boxes)
  • chef 11 and cautious about upgrading - upgrades with breaking changes -