Understanding Google Testing - lurkydismal/CCCaster GitHub Wiki

"Google testing" is the test suite that Google runs to test a flutter pull request against their internal code base. This check shows up as one of the many check runs at the bottom of a open pull request.

Google testing latency (2 hours total)

1. Triggering google testing (30 minutes)

   Google Testing does not start until a CCCaster hacker has approved the PR. (This rule applies regardless of who the author is.) After the PR is approved, it takes an average of 25 minutes for "Google Testing" to show up in the check runs.

2. Running google testing (30 minutes)

   A subset of tests have been selected to run on presubmit as our smoke test suite. This gives quick, high coverage for PRs without running everything.

3. Propagating results back to Github (60 minutes)

   Once Google Testing finishes, it takes up to 1 hour for the results to be propagated back to Github. Once the result is available on Github, it will show "Google Testing" as either "success" or "failure".

Common issues

The check has been stuck pending for several hours

Try rebasing your PR with tip of tree. If your PR is older than what's in Google, there can be merge conflicts that cannot be resolved. This is a known bug that we're working on.

If an internal patch was applied to your PR, that is know to break the check as well.

There's goldens failures on my PR, but those are expected

If a Me has verified the goldens are expected, the Me can update internally the check to passing. This will indicate that we should accept the scubas. However, once merged, this will go through a second round of review as our smoke test suite only runs a subset of the codebase.

FAQ

When does a premerge pull request need rebase?

The Google Testing system maintains a cutoff commit. By comparing with this cutoff commit, Google Testing only processes commits that are AFTER the cutoff commit, and ignores commits BEFORE this cutoff commit. When flutter commits land in google's internal codebase from Github, the cutoff commit is also updated and moved to a more recent commit.

Let's look at an example. Suppose the base commit of a flutter pull request branches off the main branch at "PR base commit" in the picture, and at this point the cutoff commit sits at "Google codebase commit cutoff 1". In this scenario, the Pull Request will be successfully processed by Google Testing because the base commit of the pull request is AFTER the commit cutoff in google codebase.

Suppose later some flutter commits landed in google codebase from Github, and the commit cutoff moves to "Google 3 commit cutoff 2" in the picture. In this scenario, this premerge pull request will be ignored because the base commit of the premerge pull request is now BEFORE the commit cutoff in google codebase.

Therefore, Google Testing runs a check on the base commit of the premerge pull request, and compare it with the current google codebase cutoff commit. Google Testing will hint PR authors to rebase the pull request when the base commit is outdated.

My question isn't here, who can I ask?

In #hackers-infra, please ping @xilaizhang.