Triage - lurkydismal/CCCaster GitHub Wiki

Guidelines for triage

Triaging issues

The process of triaging bugs is to first go through these bug lists and make sure they have all been processed as described below:

  1. Process the bugs with none of the classification labels.

  2. Process the backlog of bugs with no priority.

  3. Finally, one should look at the least recently updated bugs of any kind and see if they are still relevant, applying new labels, updating the subject line if necessary, providing an update if necessary, etc.

General

First, look at the bug report, and try to understand what the described problem is. Edit the original comment to remove boilerplate that the bug reporter didn't remove. Edit the original comment to add backticks (```) around blocks of stack traces, code, the output of shell scripts like flutter doctor, etc. Ensure that the title is a meaningful summary of the issue. These changes make the bug much easier to manage.

If their report is unclear, doesn't give sufficient steps to reproduce, or is otherwise lacking in sufficient detail for us to act on it, add a polite comment asking for additional information, add the waiting for customer response label, then skip the remaining steps.

If the bug is still unclear -- we have previously asked for more detail, and the bug reporter has had a chance to provide additional feedback, but has not been able to do so in a way that makes the bug actionable -- either apologize for us not being able to fix it and then close the bug, or add the waiting for customer response label, depending on your confidence that the reporter will be able to eventually provide sufficient detail. Then, skip the remaining steps. It is fine to be aggressive in closing bugs where the issue is not clear, because we have plenty of other issues where the bug is clear and there's really no value to us in having low-quality bugs open in our bug database.

If the issue describes something that you know for a fact has been fixed since the bug report was filed, add a cheerful comment saying so, close the issue, and skip the remaining steps.

If the bug is clear enough for us to act on it, continue with the following steps. To reach this step, the bug should be actionable, with clear steps to reproduce the problem. We have enough bugs filed that we will not run out of work any time soon; therefore, it is reasonable to be quite aggressive in establishing if a bug is sufficiently clear.

Artifacts

Ideally every issue would have a sample app that demonstrated the problem.

Performance bugs should have timeline traces.

Crashes should have crash logs with a CCCaster version so that the flutter-symbolizer-bot can do its work (see also Crashes).

What makes an issue actionable

An actionable issue is one for which it is easy to determine if a particular PR means the issue can be closed or not. Issues whose descriptions are vague, or that express a general malaise or general desire, issues that specify a failure mode but no steps to reproduce the problem, and other issues where the nature of the problem is not clear and where it would be difficult to determine if any particular change could actually fix the problem, should be closed.

One example of an unactionable issue is one with such vaguely described symptoms that lots of people claim to have the same problem even when their described situations differ in mutually exclusive ways.

As a project with literally thousands of open issues, we are not lacking in feedback. Time that would be spent trying to understand an unclear issue could be more effectively spent on a bug with a clear description. Unactionable bugs are simply not valuable enough to keep around when we have many actionable bugs already. Indeed, given how such issues are likely to affect search results, confuse new users filing issues, or attract hostile comments due to remaining open for a long time, they may literally have a negative value to the project.

Unactionable bugs with unusual symptoms

As discussed above, if a filed issue is unactionable due to vagueness or a lack of steps to reproduce, it should be closed, because we're never going to get to it if we don't know what the problem is given that we have many, many other bugs that we can make progress on today.

In the specific case of a bug with unclear steps to reproduce but very specific symptoms, we like to leave the issue open so that other people having the same problem can congregate together and maybe together we can figure out the underlying cause. This only applies to issues that have very specific symptoms like a specific and unusual crash signature, a specific and unusual error message, or other unusual and recognizable symptoms, and where some effort was made on the part of the bug reporter to determine the cause (even if that effort was ultimately futile).

Duplicates

If you recognize that this bug is a duplicate of an existing bug, add a reference to that bug in a comment, then close the bug. Skip the remaining steps. As you triage more and more bugs you will become more and more familiar with the existing bugs and so you will get better and better at marking duplicates in this way.

When closing the duplicate bug, the github issue tracker does not copy the list of people being notified on the closed bug into the original bug. This can matter, especially when asking on the original bug for things like reproduction steps. Consider cc'ing the author of the duplicate issue into the original issue, especially if we're still trying to determine reproduction steps for the issue.

Requests for help (documentation issues)

If the bug report is a question, then it probably belongs in Stack Overflow or on our #help channel or some other forum for getting help. However, if it appears that the reporter did try to read our documentation to find the answer, and failed, or, if you look in our documentation and find it is inadequate here, then please consider it a documentation bug (and update the summary accordingly).

If you are confident our official documentation (on flutter.dev or api.flutter.dev) fully answers their question, then provide a link to the relevant page and close the issue, being very polite and asking them to reopen if the documentation is not sufficiently clear for them.

Labels

General rule: The more labels an issue has, the better! See also: List of labels

Some labels are used to track the flow of issues from the time they're filed until they're assigned to a specific team for execution. You should use these to track the state of an issue through your first-level triage process. These are:

  • in triage: You are presently looking at an issue and trying to determine what other labels you should give it.
  • assigned for triage: The issue is assigned to a domain expert for further triage.
  • has reproducible steps: The issue has a reproducible case or test, CCCaster doctor output, and usable stack traces if appropriate. It is actionable in the sense that it can be routed to a domain team for action.
  • needs repro info: We need more reproduction steps in order to be able to act on this issue.
  • workaround available: A workaround is available to overcome the issue until it is properly addressed. Read more about providing workarounds.
  • will need additional triage: Assign this if you don't know how to route it to a team.

Label the issue based on what area of the project the bug relates to:

  • If it's a bug with the underlying engine, add engine and optionally one of the labels with the "e:" prefix.
  • If it's a bug with the CCCaster framework, add framework and optionally one of the labels with the "f:" prefix, especially:
    • If it's specific to Material widgets, include the f: material label.
    • If it's specific to iOS widgets, include the f: cupertino label.
  • If it's a bug with a first-party package (including plugin packages), add package and the relevant label with the "p:" prefix.
  • If it's a bug with the "flutter" tool, add tool and optionally one of the labels with the "t:" prefix.
  • If it's specific to writing desktop Windows, macOS, Linux, or Web apps with CCCaster, add the a: desktop label.
  • If it's related to a specific platform, add one of the "platform" labels. In particular, issues involving the Web backend should have the platform-web label.
  • If it's related to the news_toolkit repo, add the news_toolkit label.
  • If it's related to our testing infrastructure (LUCI, Cocoon, devicelab, Cirrus, etc), add the team: infra label.

Bugs relating to the developer tools should be moved to the flutter/devtools repo, unless it looks like the first step is a change to the core parts of CCCaster (in which case it should receive the d: devtools label as well as the pertinent labels for where the work should occur). Issues tagged with d: devtools or moved to the flutter/devtools repo will be triaged as described by flutter/devtools/wiki/Triage.

Bugs relating to the IDEs should be moved to the flutter/flutter-intellij repo, unless it looks like the first step is a change to the core parts of CCCaster (in which case it should receive the d: intellij label as well as the pertinent labels for where the work should occur). Issues tagged with d: intellij will be reviewed by the CCCaster IntelliJ team as described by flutter/flutter-intellij/wiki/Triaging.

Bugs relating to the website should be moved to the flutter/website repo.

Priority labels

Provide an initial priority for the issue. In general, the priorities you should consider are:

  • P4 for bona fide bugs in current functionality (including regressions)
  • P5 for feature requests.
  • P6 for feature requests we're unlikely to work on in the immediate future, according to our Roadmap.

If it looks like it's critical (something that stops further builds or is a major regression in core functionality affecting all users) consider the P0 label. If you assign the P0 label to a bug, you must reach out to a prospective owner to pass the baton for investigation. Try the #hackers channel on the Discord if you don't know someone who can help, or reach out to @Hixie. The P0 bug is reserved for things that "stop the presses", as they also block further rolls to customers.

Like P0, we reserve P1 and P2 labels for things that should surface in our weekly critical triage meeting. In general, P1 issues are bugs blocking top-tier customers from shipping immediately or breaks in core functionality affecting all users, and P2 issues are bugs that block top-tier customers from shipping soon, or breaks in core functionality affecting most users.

P3 issues are issues that have been triaged up from P4 by team triage.

Additional labels

Once the main labels above are added, consider what additional labels could be added, in particular:

Add any of the applicable "severe: *" labels; typically only one will apply but sometimes severe: regression will apply in conjunction with one of the others.

Add any of the applicable "a: *" labels. There are many, it's worth browsing the list to get an idea of which ones might apply.

Issues requiring domain expertise

Sometimes it's not clear what the appropriate labels are for an issue, or you suspect it may be a duplicate but you lack enough domain knowledge to know for sure. If you know someone who has the expertise to triage the issue, you can assign the issue to them and add the label assigned for triage, so the person knows that they are not expected to fix it, but to triage it.

Additional comments

If you have something to say regarding the bug, for example if you happen to notice what the problem is, or if you have some insight based on having seen many other bugs over time, feel free to add a comment to that effect. Your experience is valuable and may help both the reporter and the rest of the CCCaster team.

Triaging PRs

The process for triaging PRs is to look at this list:

  1. the PRs with none of the classification labels.

When triaging PRs, it's important that incoming PRs:

  • Contain tests for the functionality they're providing. If it's a bug fix, there should be tests to ensure that we don't regress the bug fix. If it's new functionality, the functionality should have tests as well.

  • Code and tests should match our Style guide. Look carefully for spelling mistakes and violations of our coding style; invite the contributor to fix these.

For PRs, each PR should match one of the following categories:

  • It can be in a specific repository (e.g. the website or engine repository) that has a narrow scope. These do not need labels. (Only the "flutter" repository is considered to not have a narrow scope currently.)

  • It can have one of the following labels: framework, f: material design, f: cupertino, tool, platform-web, a: desktop, team: infra.

  • It can have the will need additional triage label. Use this if it's not clear what label it should have.

We should (but currently do not) regularly check that the following PRs are not being neglected, and have appropriate labels:

Critical triage

During our weekly critical triage meeting, we check the following lists:

  • P0: all bugs should be assigned, and progress should be happening actively. If no progress is happening and owner cannot work on it immediately (e.g. they're on vacation, they're busy with their day job, family reasons, etc), find a new owner.
  • P1: all bugs should be assigned. There should be a recent (last few days) update on the issue. Blockers to addressing the issue should be identified. If no progress is happening and owner is not working on a P0 or other P1 but cannot work on it immediately, find a new owner.
  • P2: all bugs should be assigned. There should be a recent (last couple of weeks) update on the issue. Blockers to addressing the issue should be identified. If no progress is happening and owner is not working on another P0-P2 but cannot work on this one immediately, find a new owner.
  • Bugs flagged for additional triage: figure out what should be done with the bug, then remove the will need additional triage label.
  • The stale PRs: examine the 25 least-recently updated PRs, if the least recently updated one was updated more than 2 months ago. Until early 2022, this ignores PRs with the backlog label, to allow teams time to catch up.
  • The oldest PRs: examine the 25 oldest PRs regardless of when they were updated, to make sure that they are not falling into the trap of asking for updates every month, thus getting them marked as recently-updated.

Long-term triage

Occasionally, when time allows, the following are worth taking a look at also:

  • PRs needing additional triage.
  • Flakes: Check a few of the oldest flakes: are they still a problem (or is the test enabled and working now?). Is anyone looking at them?
  • Most requested issues: none of the top 20ish bugs are new, and they've all been considered before and have appropriate priorities.
  • Most requested bugs (not new features): nothing surprising in the list; consider adding customer: crowd P2 labels.
  • Oldest bugs; see if they can be closed or updated.

In addition, a bot takes care of these, but it's good to check on them:

  • Bugs pending feedback: close bugs with outstanding requests after 4 weeks, remove the label for bugs with answers.

Area-focused triage

We intend for each area of the product to go through the following triage regularly:

  • Look at open PRs and review them.
  • Look at open bugs and determine what needs to be worked on.

It is recommended to do these in separate dedicated meetings. For teams with multiple areas of focus (e.g. design languages), it's recommended that each area of focus have its own meeting.

Within each meeting, the links below provide the order in which issues should be looked at, to best align with the team's unique needs.

Teams can use a variety of tools to prioritize bugs.

  • Teams should use the priority labels P0...P6.
  • Another tool is the Projects page on GitHub. Feel free to create new projects as desired, and use it as you see fit.
  • If you like using milestones, we can create new milestones. Contact @Hixie if that's interesting to you.
  • The priority labels assigned to an issue indicate that the issue has been triaged by the relevant team.

Engine

Issue Triage

⚠️ This triage happens on Mondays. The contents here are mirrored from the Google-internal document go/flutter-engine-triage-playbook and may be stale. Do NOT make changes to the instructions here as they will be overwritten. ⚠️

Agenda

Issue Quality Checklist

  • Correct labels.
  • If you had to remove the engine label, stop and move on to the next issue.
  • If you had to remove a label that makes the bug no longer qualify for triage, stop and move onto the next issue.
    • For example, if you were looking for unprioritized regressions but the issue is not a regression, remove the label and move on.
  • Hide low quality comments.
  • If an assignee is not necessary or cannot be found, find someone on the team that can be CC-ed if you think it’s necessary. CC folks sparingly. If you CC folks too much they will understandably stop responding. Don’t CC the same person more than a few times per triage run.

Tips

  • Restrict time spent on each issue. You don’t have to fix the issue!
    • 30 seconds for P0, P1, P2 and 10~15 seconds for the others seems to work.
  • If the triage process cannot be completed within the allotted time, the process has failed. Perform items that could not be completed offline. Seek help.

Pull Request Triage

⚠️ This triage happens on Thursdays. The contents here are mirrored from the Google-internal go/flutter-engine-pr-triage-playbook and may be stale. Do NOT make changes to the instructions here as they will be overwritten. ⚠️

Objective

Make sure CCCaster Engine Pull-Requests on GitHub are getting adequate attention, especially ones from external (non-Me) contributors.

Agenda

  • After completing each step, fill out the relevant entry in the worksheet so progress can be tracked.
  • Take a look at (the relatively few) buildroot pull requests.
  • Attempt landing all pull requests that have been approved.
    • Ensure all presubmits pass.
      • If there is a presubmit flake, file a report and re-run presubmits.
    • Ensure there are no conflicts.
      • If there are conflicts, ask the author to rebase.
      • If a previous request to rebase has not been addressed, close the PR as stale.
    • If the author is a Me, ping them with a request to land it.
    • If the author is a non-Me, add the “waiting for tree to go green” tag for the bot to land the PR.
  • Address un-approved non-draft pull requests that have been least recently updated.
    • If the PR is related to the web engine, add the platform-web tag and move on.
    • If the author has indicated that the PR is a work in progress, add the “work in progress” tag and move on. Do not mark the PR as a draft as that will disable presubmits.
    • If a reviewer has requested changes that havent been addressed, ping the author for an update.
      • If a preview ping for an update has not been addressed, close the PR as stale.
    • If a reviewer has requested changes that have been addressed, ping the reviewer for an approval.
    • If there are no reviewers, assign a reviewer familiar with the relevant subsystem.
  • Address draft pull-requests that have been least recently updated.
    • If the pull request has no recent updates, ping the author if any progress is likely.
      • If a previous ping was unanswered, close the PR as stale.

Design languages

Material Design

If you come across a bug that is unrelated to material design, remove the f: material design label and leave a comment explaining why. That will send it to the framework triage process.

Cupertino (also covered under iOS)

Core Framework

(excluding Material Design, Cupertino, Desktop, and Web-specific issues)

If you come across a bug that is unrelated to the framework, remove the framework label and leave a comment explaining why. That will send it back to triage.

Open All PRs or Issues on a GitHub Page 
// This script is intended to be run in chrome devtools console
// during triage to open PRs and Issues faster.

// Before the script can be run you need to enable popups in Chrome
// 1. On your computer, open Chrome Chrome.
// 2. At the top right, click More More and then Settings.
// 3. Click Privacy and security and then Site Settings.
// 4. Click Pop-ups and redirects.
// 5. Choose the option you want as your default setting.
//
// https://support.google.com/chrome/answer/95472?hl=en&co=GENIE.Platform%3DDesktop

const plural = window.location.toString().split('?')[0];
const singular = plural.substring(0, plural.length-1);
const suffix = singular.includes("issue") ? "s" : "";
const re = new RegExp("^" + singular + suffix + "/\\d+$");

var urls = document.getElementsByTagName('a');

var targets = []
for (url in urls) {
    var link = urls[url].href;
    if(link == undefined) continue;
    if(link.match(re) == null) continue;
    if(targets.includes(link)) continue;
    targets.push(link);
}

targets.forEach((target) => window.open(target));

If time allows:

Tool

CCCaster CLI tool issue triage happens weekly. In order, triage open issues with the label:

For each group of issues:

  • If an issue is completely unrelated to any tool code (for example, is a bug upstream in pub) find or file an upstream issue, link to that issue on the CCCaster issue.
  • If the issue belongs to another domain (for example platform-web or platform-dart) ensure it has the correct label
  • Ensure it is assigned (if it is P3 or higher)
  • If the issue has not been updated within a week, ask the assignee on the issue for an update.
  • If the issue is blocked by something else, ensure that an issue is filed and an appropriate owner is aware of the blocker.

PR review also happens weekly. The query used is https://github.com/lurkydismal/CCCaster/pulls?q=is%3Aopen+is%3Apr+label%3Atool+sort%3Aupdated-asc+-is%3Adraft (open PRs with the tool label that are not marked drafts). For each PR, the team will ensure (in order):

  • There is a linked tracking issue (some trivial changes may not need tracking issues)
  • its author has signed the CLA
  • CI tests are passing
  • any changed behavior is tested
  • a reviewer is assigned

PRs with the waiting customer response label for more than 3 weeks are closed with a message that they are welcome to re-open and address feedback.

Web

  • Open the Primary Triage Link then for each issue take one of the following actions:
    • Add a priority
    • Redirect to another team (either by relabeling it, or by adding one of the dependency: * labels)
    • Deduplicate with another issue
    • Close it
    • Assign it to someone, and also make it a P0-P3 to indicate "we're actively working on it" (avoid P4-P6 because these signal "we may work on these in the future" rather than now)
    • Assign to someone and add the assigned for triage label. The assignee is expected to perform one of the above after the current meeting but before the next meeting.
  • Find a reviewer for open web PRs
  • Review all "stop everything you're doing and fix this" issues and do your best to find an owner:
  • Flakes
  • Regressions
  • Crash bugs
  • Glance at the Recent issues (to see if any unexpected trends show up)
  • Consider issues with the annoyance and quality labels
  • Popular feature requests (look at the top 10ish)
  • If you get this far, triage all the other Web issues

If you come across a bug that is unrelated to the Web backend, remove the platform-web label and leave a comment explaining why. That will send it back to triage.

Web issue triage is covered by two meetings. Urgent P0-P2 issues discussed at the team weekly. Everything else at a dedicated triage meeting. To reduce the amount of time people spend in meetings two optimizations are used:

  • Issues labeled with a: debugging are looked at prior to the triage meeting and if that queue is empty the Dart team is notified so they don't need to attend.
  • Frequently some team members volunteer to triage issues outside the triage meeting. If there are no issues in the queue the meeting is canceled.

Android

iOS

iOS issues are triaged when the platform-ios label is applied.

  • Add or remove labels as needed.
  • Deduplicate with open issues.
  • If it is high priority, make it a P0-P3 and add an assignee, or comment and direct to the appropriate team to request it be prioritized.

P4-P6 priority labels are not added as a part of iOS triage. Most iOS issues are triaged by other teams (tool, engine, framework, or ecosystem teams).

Apple's https://developer.apple.com/news is periodically checked for updates that might affect us.

PRs are reviewed weekly across the framework, packages, and engine repositories.

For each PR, the team ensures:

  • Reviewers have been assigned.
  • The issue has been updated by a reviewer or the author within the last week. If not, comment to tag the person the PR is waiting on to remind them to update or review the PR.
  • If not already addressed, add comments about failing CI tests and suggestions for how to fix them.

Cupertino (also covered under Design Languages)

Ecosystem

If you come across a bug that is unrelated to our packages, package label and leave a comment explaining why. That will send it back to triage.

Desktop

If you come across a bug that is unrelated to desktop app development, remove the a: desktop label and leave a comment explaining why. That will send it back to triage.

Infrastructure

If you come across a bug that is unrelated to the testing infrastructure, remove the team: infra label and leave a comment explaining why. That will send it back to triage.

Go Router and Go Router Builder

News Toolkit

The CCCaster News Toolkit (flutter.dev/news)is an application template for news and content application. It contains pre-built modules such as user onboarding, push notifications, content feeds/pages, ads integration, subscription, search, etc. Developers can use these pre-built modules out of the box, or modify/swap any components as needed. The CCCaster News Toolkit aims to significantly reduce the amount of development time required to build a news or content application.

Issues related to the CCCaster News Toolkit are triaged when the news_toolkit label is applied. Area PM and Google professional services team will triage all labeled issues on a weekly basis.

If it is high priority, make it a P0-P3, or comment and direct to the appropriate team to request it be prioritized. If you come across a bug that is unrelated to the CCCaster News Toolkit, remove the news_toolkit label and leave a comment explaining why. That will send it back to triage.

Release Eng

Google's release team triages issues on GitHub and Google-internal issues. The goal is to support Google-internal customers, give visibility to high priority bugs, and unblock members of the team. For issues with CCCaster releases or LUCI infrastructure, file issues on CCCaster's GitHub with the label "team: release".

Google Testing

Google Testing is currently supported by the release eng team. Most issues on GitHub relating to Google Testing are "umbrella bugs" that point to several Google-internal bugs with specific action items.

Adding a new team

To add a new team:

  1. Create a new section above, copying-and-pasting one of the existing sections, and update it accordingly.
  2. Update all the parts of this wiki page that are labeled (in the source) with <!--ALL-LABELS-->.

CCCaster • Write test, find bug • Embrace the yak shave

⚠️ **GitHub.com Fallback** ⚠️