Verigreen Operation - Verigreen/verigreen GitHub Wiki

Verigreen UI

Accessing Verigree Web UI is done via the URL defined by your deployment process, in the "Verigreen Deployment" Wiki page.
Generally speaking, Verigreen UI is covering all that happened in your protected branch. All failed jobs , triggered failed or merge failures are recorded and shown.
The UI shown below is a typical high pace team flow, in which you can see almost all states an item may be in. we will go over briefly on the important ones, as the rest are straight forward ones and understood:
alt text
Let's start with a general description of the data structure you see in the UI:

  1. Each line item seen represents a commit push made to the protected branch. Its name, under the "Id" column, consists out of "vg" prefix, the git commit ID and the committer name.
  2. Column 2 is the protected branch, the original target branch the commit was aimed to. In this example, this is the master branch. In case you have more than one protected branch, you will see other names as well, depending on the commit's target branch.
  3. Parent branch - where the verified commit was branched off. It can be the protected branch itself, or another in process commit branch in case a commit arrived to Verigreen during a verification of a previous one. As can be seen from the image above, commit Item ID vg_19e63a4_... was branched off vg_54fccbf_... rather than from master, hence we say vg_19e63a4_... is a child of vg_54fccbf_... This is important to note since in case of a child branch's job completes its running successfully before the parent one does, the parent job is being aborted by Verigreen since the code was verified and pushed already by the child job.
  4. The committer name - the person who made the commit. This is not necessarily the person who pushed the code to the remote git repository.
  5. Status - the is the column which shows the commit Item current status. There are several states the commit item can be in and we will elaborate on that later on. For now, the most important ones the user will usually see are these:
  • NOT_STARTED - Verigreen didn't start processing the request
  • RUNNING - Verigreen is now processing the request
  • PASSED_AND_PUSHED - the commit Item was verified and pushed successfully to the original remote target branch it was aimed for
  • PASSED_BY_CHILD - the commit was indirectly verified by a child commit. It was not necessarily pushed to the remote repository yet
  • FAILED - verification has failed. Could be due to a build break, network issues, test failure and such.
  • TIMEOUT - the verification process did not complete by the time allowed for it in the configuration file
  • TRIGGER_FAILURE - Verigreen was not able to trigger the verification job. Could be network issue, permission issue on the server side, etc
  • MERGE_FAILURE - Verigreen was not able to merge the new commit on top of the parent branch. Usually happens at the beginning of the process so verification is not triggered
  • GIT_FAILED - Git has failed. Could happen due to a merge failure at the end of the verification process, sync problems between the git client and the remote, etc
    There are a few more statuses and we will refer to them later on in the document.
  1. Retry mechanism - if enabled (via configuration file), allows anyone to rerun a verification process if it failed one or more times. The button is grayed out in case of any other state of the commit item. Currently there is no limit to the number of times Verigreen allows you to do it. However, using this option too many times on the same commit may cause an increase in feedback time from the CI system since new verification processes are being spawned at each press of the button
  2. Creation time, run time and end time - refer to the verification job
  3. Build URL - a link to the verification job in your CI system (Jenkins in this case)
    The search box on the left side is a dynamic one. By that we mean that the search results are updated as you type your query.
  4. History - if you have a commit which has a history associated with it (has been verified more than once), a green '+' (plus) sign will be visible on the left hand side.
    Clicking it will expand the history to reveal all the instances this commit has been part of: alt-text In this screenshot you can see that the original commit originated not from the protected branch (master" but from a running branch. It failed ('a+b') so a new permutation of the commit from the protected branch ('b') started and actually passed.

Impact on developers

When a developer pushes his code, a message is being displayed, alerting him that his code was temporarily rejected and is now being Verified by Verigreen. This message appears regardless of the git client you are using, unless you somehow suppress remote messages. The message looks like The following: alt-text
The reason for the reject message is that we can't (and shouldn't) deceive the git client by behaving as if the change has reached its destination. This will get the system out of sync and might be dangerous to all. Instead, we say exactly what is happening: your code has been submitted for verification by Verigreen. Once the verification has ended, the committer will be notified on the status - success or failure. The "reject" message at the end can be ignored by the developer if it's accompanied by the "Submitted for verification by Verigreen" message. After this message, the developer can continue working as usual (continue writing code, commit and even push a new set of changes to the remote branch) while waiting for the verification results.

Verigreen and Jenkins

verigreen currently supports Jenkins solely as a verification CI system. This may change in the future but it is yet to be determined in what way.
One of the directions considered is a more general approach, in which any RESTful CI system might be an option for a verification system.
Currently Jenkins is the most popular CI/CD solution in the industry, so as such it suits our needs very well for a gated check-in system solution.

Verigreen is built in such a way so it does some optimizations regarding running the verification jobs:

  1. Verigreen first attempts to merge the coming changes on top the current HEAD of the protected branch (on a separate, temporary branch). Failing to do so for any reason will fail the process and no jobs will be triggered.
  2. When a new commit triggers a job, if there are any running verification jobs at that time, Verigreen will branch off the latest commit in the system (even though the current verification process isn't finished yet) and will trigger a new verification instance (subject of course to the number of executors on the relevant node + concurrent build option is enabled).
    Since the latest commit also includes all preceding commits, in case the latest commit verification process completes before its parent commit does (faster build machine, for example), all parent jobs are cancelled, freeing executors for additional processes.
  3. If commit 'a' is running, and while it is running commit 'b' enters and built upon a (as in 'a+b') , in case 'a' fails while 'a+b' still runs, Verigreen cancels 'a+b' verification job and spawns a new verification of 'b' only, which doesn't include 'a'.
    This will apply to any number of child processes currently running in your system. That is why you may be seeing jobs being aborted in your CI system during verification. The assumption here is that commit 'b' doesn't include a fix for commit 'a', hence chances are 'a+b' will fail due to the failure in commit 'a'. This behavior might change in the future - we're considering adding a configurable policy of whether to try and run 'a+b' if 'a' fails (at the expense of additional potential unnecessary job in Jenkins).