Test Kitchen and CI - chef-boneyard/chef-summit-2014 GitHub Wiki

Thursday, Greenwood, 1330

How do you solve a combinatorics problem of multiple drivers, multiple platforms, and multiple suites with Test Kitchen? Can you spin up multiple thread for each of these to run these in parallel?

Kitchen Matrix

Can you publish a community cookbook to validate with Test Kitchen?

.kitchen.cloud.yml

-- Can set an environment variable so you can pass in your cloud credentials from local.

In a perfect world all public cookbooks would have something reasonable, perhaps a vagrant driver. Anything else could be handled in a .kitchen.local.yml.

A Jenkins plugin to pickup different platforms out of the .kitchen.yml, run them all, and spin them up on Jenkins workers seperately.

Error Ranges for errors that Test Kitchen throws?

ie:

If an error happens for the instance creation phase it has an error range:

For example: If it's a transient error that occurred during instance creation? Throw 11 Transient error converge: Throw 21 Transient error busser: Throw 31 Transient error busser: Throe 41

-- The idea is that you could ignore errors that are in different phases, or retry those errors, while in certain phases it's very bad to throw those errors.

Being able to spin up nodes in a number of minutes as a failure state? Other Metadata in the Test Kitchen run:

Amount of time it took the Kitchen run to take? What is the result of the chef run? Test framework?

How can you make an AWS cookbook test portable?

Prehook and posthook in Test Kitchen? Return values after a test kitchen run so that the next action can take.

Setting up objects for Test Kitchen? We're not just running nodes these days:

• Objects in AWS
• EBS volumes
• Create and destroy AWS instance

-- Simian Army pattern for AWS to clean up unused images and whatnot

kitchen test --destroy=Success

"CI is different than development.": Success: Destroy instance on Success of the verify Always: Always destroy the instance --> Make sure "Always" bubbles up if an exception is handled.

How do you handle artifacts and build?

• Try to do your CI in the same environment that you do production on
• Your build artifacts: Do what you do in production
  • Berkflow for making artifacts to package cookbook artifacts
  • If you resolve dependencies directly you can just use Supermarket
  • Your results might vary depending on your business (regulartory compliance vs Hey fuck it we can just push to production and it doesn't matter if we brake 1% of machines.)

If you need it, there's a concurrency flag on kitchen:

kitchen test -c 10:

• By the way, if one fails, they all fail.
• Really hard to make sense of the output. -- Each instance has it's own dedicated file logger.
• How can we set different failure strategies for concurrency?? (Fail fast vs report everything).