This document attempts to outline the steps that are needed to cut a release and then get it deployed on the production environment. It is accurate as of the 13th of March 2015

Cutting the tag

Hilary

On release0 do:

• cd /opt
• rm -rf oae
• puppet agent -t
• stop the hilary service: service hilary stop
• run the tests in /opt/oae with grunt test (ensure that there are no failures. If they are you should be able to explain thoroughly why that is the case)
• clear out the /opt/oae directory (things like log files, target, previous releases, ..): git reset --hard HEAD
• run the release script (bin/release -s -r origin -v 10.1.1)
  • It will add a commit that:
    • bumps the version in package.json
    • run npm shrinkwrap
  • It will create a tag that points to the new commit
  • It will add a commit that takes the shrinkwrap file back out so the latest dependencies can be pulled in during development
  • It will push the commits and tag to GitHub At this point you will have to provide your GitHub username and password
• Once everything is pushed to github, TravisCI will do its thing. See TravisCI section below for continuation on Hilary

3akai-ux

The release process for 3akai-ux is very similar, on release0:

• cd /opt
• rm -rf 3akai-ux
• puppet agent -t
• stop the hilary service: service hilary stop
• cd /opt/3akai-ux
• run npm install
• run bin/release -s -r origin -v 10.1.1
  • It will add a commit that:
    • bumps the version in package.json
  • It will create a tag that points to the new commit
  • It will push the commit and tag to GitHub. At this point you will have to provide your GitHub username and password
• Once everything is pushed to github, TravisCI will do its thing. See TravisCI section below for continuation on 3akai-ux

Travis CI

For both Hilary and 3akai-ux, travis will:

• Run the unit and integration tests (if there are known intermittent failures in the tests, you might want to replace the test script with /bin/true)
• If successful:
  • (UI only:) Minify all static assets
  • Create a tarball that includes the code and all (node) dependencies
  • Upload the tarball to Amazon S3 to the oae-releases bucket. The generated tarball will include:
    • The tag name (e.g., 10.1.1)
    • The node version on which the tests were run (e.g., 0.10.30)

Verifying releases on QA

On QA's puppet master node we configure a single QA machine (say qa0) to test run the release by configuring environments/qa/hiera/qa0.json with:

"app_install_method": "archive",
"app_install_config": {
    "version_major_minor": "10.1",
    "version_patch": "1",
    "version_nodejs": "0.10.30"
},
"app_ui_path": "/opt/3akai-ux/optimized",

"ux_root_dir": "/opt/3akai-ux",
"ux_install_method": "archive",
"ux_install_config": {
    "version_major_minor": "10.1",
    "version_patch": "1"
}

Go to QA0 and run the nightly (/opt/scripts/nightly.sh) or do the following:

service hilary stop
service nginx stop
cd /opt
rm -rf 3akai-ux
rm -rf oae
puppet agent -t

Test. Now test it again.

Rolling releases out on production

Read the Upgrade Guide notes for the new release and make sure you know if there's anything that needs to happen before or after you roll out the release. If there are no notes, it might be worth pinging the development team whether they've forgotten to put them in the wiki.

Update the version numbers in puppet-hilary's production environment. Push your changes to the oaeproject GitHub repository. Do NOT pull it on the production puppet master node. The deploy script will do this for you.

Performing the actual roll-out can be done by:

• ssh-ing into the bastion node
• cd'ing into the oae-fabric folder
• running fab cluster.oae.upgrade

This will:

• Try upgrading a single node that isn't user-facing (pp0)
• If all goes well it will:
  • Upgrade all other non-user-facing app nodes (activity, pp)
  • Upgrade half of the REST serving nodes (app0, app1)
• If all goes well it will:
  • Upgrade web0
• If all goes well it will:
  • Upgrade the remaining REST serving nodes (app2, app3)

It's usually a good idea to tail the logs on the syslog node (in another terminal) and tail the logs to see what is going on cd /var/log/rsyslog tail -f app0-....local0.log | ./filter-bunyan

In case something goes wrong and you can't see an immediate fix for it, try and revert to the previous release. This can be done by restoring the previous version numbers in puppet-hilary and repeating the release procedure.

Update the Release Upgrade Guide in case there were any instructions under the "Next Release" section.