Debugging ocd environment webhook Config - ocd-scm/ocd-meta GitHub Wiki

Helmfile will log what failed. You might see things like missing required env vars or invalid yaml. If you fix that you might find that you cannot get a deployment due to an error message no deployed releases. This means that the previous attempt is in failed state so it cannot run a diff to see what to change. The solution is to purge the failed deployment. Make sure that you have set TILLER_NAMESPACE to name the project where helm tiller is installed then check for failed deployments with:

helm list | grep -i fail

Then hard delete the failed deployment with:

helm del --purge $FAILED

If you cannot see an obvious reason to fail and you have purged broken releases then try the following.First of all you should see what the ocd-envrionment-webhook logged on your openshift pod logs such as:

Release "slackbot" does not exist. Installing it now.
Error: release slackbot failed: BuildConfig ... Strategy: v1.BuildStrategy.SourceStrategy: ... ReadString: expects " or n, but found 2, ... {"env":[{"name":"clientId","value":249550160629.51508},

See the note about turning up the log level below for more details. Note I removed a lot of noise in that logline. The bug is invalid json "value":249550160629.51508 has missing quotes and should be "value":"249550160629.51508". That was a bug in the template not expecting '.' characters in values and forgetting to quote the string. If you don't spot the bug from the log then look for the failed release:

helm list
slackbot   1   Sun Jan 13 07:57:17    FAILED    ocd-builder-1.0.0

Next, you can dump out the yaml for the version number that is named as failed with:

helm get --revision 1 slackbot

Now you can diff with a working release or debug by trial and error using oc create to try to get kubernetes to directly apply yaml you a debugging for syntax errors or other "your yaml isn't correct" bugs.

If you need to turn up the logging level its is best to do that on the terminal so that any secrets are log sent to the openshift log archive. You can do this as:

# find the pod
oc get pod | grep ocd-environment-webhook | awk '{print $1}'
# login to it
oc rsh ocd-environment-webhook-6-ns65f

In that shell do export OCD_LOG_LEVEL=debug then run /usr/local/bin/ocd-hook.sh $REPO_REF where REPO_REF is sent by the git webhook so it is typically refs/heads/master but you can look that up on github by searching for the last webhook event and looking at the ref value in the payload. Running with debug logging it should show all the yaml that it creates for you to see what helm is rejecting. You can put the yaml into a file and run oc create -f debug.yaml to see what oc might be complaining about.