Issue Labels and Issue Management - ESCOMP/CTSM GitHub Wiki

Table of Contents

General notes for collaborators

Much of this wiki page is aimed at the core CTSM development team. However, here is some brief guidance for collaborators:

We use issues in CTSM not only to track bugs, but also to track and discuss planned enhancements to the model. These enhancements include new science capabilities, improvements to existing science capabilities, and improvements to the model's software infrastructure. We strongly encourage collaborators to open an issue in the early stages of a project in order to get feedback from other CTSM scientists and software engineers and to avoid duplicated work.

For more guidance on opening an issue like this, see the quick start to CTSM development with git.

Issue management / triage

Check every week

Review issues labeled next

Every week, we should review any outstanding next issues to see what near-term priorities still need to be done (or if there are any next issues that should have that label removed) (note: this search also includes PRs):

is:open label:"tag: next"

Check whether there are critical bugs that should be added to the "high priority" project

Many (though not necessarily all) bugs with the "critical" tag should be given the "priority: high" label and added to the "high priority" project. We should frequently check to make sure none of these fall through the cracks, via this search:

is:issue is:open label:"tag: bug - critical" -project:ESCOMP/CTSM/25

Review high priority issues

First, check whether there are any issues newly-labeled "priority: high" that are not yet in the "high priority" project board, via this search (note: this search also includes PRs):

is:open label:"priority: high" -project:ESCOMP/CTSM/25

Add any resulting issues to the board. (One way to do this is from the project board: Click on "add cards", then search for: is:issue is:open label:"priority: high"; add any resulting cards to the board.) (Note that the important thing for issue triage is that an issue be on this board. The label is mainly useful as a visual indicator that an issue is high priority when viewing a big list of issues. So it's okay if an issue is on this board but doesn't have the "priority: high" label.)

Then review the High priority project board; make sure issues are prioritized appropriately and are not falling through the cracks.

Review project boards for any upcoming releases

We should have a project board for each upcoming release. For releases coming up soon, review the project board weekly; for releases further out, we should review the project board every few weeks.

Review upcoming tags

Review the Upcoming tags project board

Periodically review and triage the issue backlog

Search for issues that should be reviewed regularly

A starting point for this search is:

is:issue is:open -label:"priority: low" -label:"blocked: dependency" no:project

On top of that, we will often filter on specific types. Typically, we'll pick one type (e.g., "bug") and take as many weeks as is necessary to go through all issues of that type before moving on to the next type.

Notes on what this excludes

This excludes:

  • Low priority issues: these should be periodically reviewed, too, but at a lower frequency

  • Issues that are blocked by other issues: these should periodically be reviewed to check whether the blocking issue has been resolved, but they don't need to be included in our standard issue triage

  • Issues in projects: these are excluded under the assumption that projects have their own prioritization / triaging process

    • Note about adding issues to projects: Adding an issue to a project removes it from our normal triage process. So, if there is an issue that you want to include in a project but also want to include in the normal triage process (typically because it is useful to resolve this issue for general reasons as well as for that one project), then introduce a note in the project that references the issue, but don't actually add the issue to the project. For example, if an issue is partly for the sake of improving performance but also partly for some other purpose, then do not add that issue to the performance project; instead, add a note in the performance project that references this issue.

    • Note about closing projects: For similar reasons, you need to be careful when closing a project. If there are any still-open issues in the project, they will not be considered in our normal triage process. Thus, when closing a project, either (a) ensure that there are no open issues left in the project, or (b) delete the project rather than just closing it.

Strategies for dealing with issues during an issue triage

During an issue triage, we aim to either increase or decrease the visibility/priority of most issues; the "no action" choice should only be done for a minority of issues. Possible actions are:

  • To increase an issue's visibility/priority, do one or more of the following:

    • Add the priority: high label and add it to the High priority project

    • Add the tag: next label, indicating that someone should look at this more in the next week or two

    • Assign it to someone (which generally indicates that that person should work on the issue relatively soon)

    • Add it to the Upcoming tags project

    • Add the tag: simple bfb label, so that it will be considered for the next "simple bfb" tag

    • Add the tag: good first issue label, so that it will be considered for new contributors to CTSM

  • To decrease an issue's visibility/priority, do one or more of the following:

    • Close it, adding the appropriate closed label

    • Add the priority: low label

    • Add the blocked: dependency label

  • Other actions:

    • Add the issue to a project; then, in the future, the issue will be considered in the planning process for that project rather than our standard triaging process; however, see the note above about adding issues to projects

    • Do nothing (this should only be done for a minority of issues)

Finding issues without a type label

All issues should have a type label. Those that don't may be missed in issue reviews.

is:issue is:open -label:"type: -discussion" -label:"type: -external" -label:"type: -investigation" -label:"type: -support" -label:"type: bug" -label:"type: code cleanup" -label:"type: documentation" -label:"type: enhancement" -label:"type: tests"

Issue labels

Issues can be given a number of different labels, in the following categories. All issues should be given one type label. Other labels are optional.

type

This label defines what type of issue it is; all issues should be given one of these type labels. We give an issue just a single type label, based on the best fit for that issue. (One reason for just assigning a single type label is that it facilitates periodically reviewing issues one type at a time: if an issue had multiple types, we would waste time reviewing it multiple times.)

  • Issues requiring code (or documentation) changes

    • type: bug: Something is working incorrectly. This may cause incorrect science results - in which case it should also be given the tag: bug - impacts science label - or may have some other impact, such as the model crashing. If this causes significant problems in important model configurations, the issue should also be given the tag: bug - critical label.

    • type: code cleanup: Code changes whose primary purpose is to improve the internal code structure (i.e., refactoring). These may result in behavior changes (e.g., answer changes due to reordering calculations), but these behavior changes are incidental rather than the main purpose of the code changes. This can also include additions or edits needed in code comments.

    • type: documentation: Additions or edits needed in user-facing documentation. This includes the user's guide, tech note, wiki, README files, and xml files. In most cases, issues with this label have no impact on the running of the code and should not require testing. However, this is not always the case: for example, changes in xml files require at least some testing.

    • type: enhancement: Code additions/modifications that bring in new capabilities or improve the behavior of existing capabilities. The changes can be of a scientific or purely software (e.g., user interface) nature; if this involves new science capabilities, the issue should also be given the tag: enh - new science label. Note that, if the issue mainly involves changing internal code structure rather than external behavior, it should be given the type: code cleanup label rather than this one.

    • type: tests: Additions or changes to tests (either unit tests or system tests). (Other issue types, such as enhancements, often involve adding tests as well. Or if a feature is removed it might include removing tests. Issues with the type: tests label are just about adding/changing tests.)

  • Issues not requiring code (or documentation) changes (at least, not initially: e.g., a "discussion" or "needs investigation" issue could lead to the need for changes later) (note: these labels have a prefix of - simply to force them to all appear together in GitHub's alphabetic sorting)

    • type: -discussion: This issue's resolution involves discussion within the github issue page.

    • type: -external: This issue really needs to be resolved in some other repository. We have an issue here just for the sake of project or task management / tracking.

    • type: -investigation: A task that involves some investigation. For example, this could involve performance evaluations. As opposed to type: discussion, there will often be significant work needed to close this issue. Once this is resolved, other issues (enhancements or bugs) may be opened based on the findings.

    • type: -support: A user or developer has run into trouble or needs support with getting something to work.

priority

This label describes where this issue should fall on the priority list - i.e., whether it should be addressed soon, or whether it can wait for a while. Issues not given a priority label implicitly have a medium priority.

  • priority: high: This issue should be addressed soon

    • Issues with this label should also be added to the High priority project board. The important thing for issue triage is that an issue be on this board. The label is mainly useful as a visual indicator that an issue is high priority when viewing a big list of issues.

  • priority: low: This issue can be deferred for a while

blocked

This label describes various reasons why an issue can't be dealt with immediately. We can periodically review issues with these labels to see if the condition blocking a given issue no longer applies.

  • blocked: answer changing: This issue would be relatively easy to resolve, but we are waiting to resolve it until we can accept answer changes on master. (Note: many answer-changing issues will NOT have this label; this is mainly used for issues for which the only thing stopping us from fixing it right away is that it changes answers.)

  • blocked: dependency: This issue depends on some other issue, so we won't start working on it until that issue is resolved. Both issues should reference the other issue somewhere in their comments:

    • The blocked issue should use the phrase Blocked by #XXX in a comment

    • The blocking issue should be given the label, tag: blocker, and should use the phrase Blocks #XXX in a comment

closed

These labels can be applied to issues that are closed without being fixed.

  • closed: duplicate: This issue is a duplicate of another issue. The duplicate issue number should be noted in an issue comment.

  • closed: moved: This is an issue with some other component, so the issue has been moved to a different repository.

  • closed: non-issue: This is not really an issue. For example, someone may have thought there was a bug, but in fact the code is operating as intended or there was user error.

  • closed: wontfix: We have decided not to fix this bug, not to implement this enhancement, etc.

tag

These are miscellaneous labels that can be applied to any issue to help people find certain classes of issues later.

  • tag: blocker: This tag should be used for issues that are blocking some other issue. The other issue should have the label, blocked: dependency. This issue should use the phrase Blocks #XXX in a comment. The purpose of this label is to help us remember to remove the blocked: dependency label from the other issue once this one is resolved; in addition, by calling out that this issue is a blocker, we may give it higher priority.

  • tag: bug - critical: This tag can be applied to bugs, and means that this bug causes significant problems in important model configurations.

  • tag: bug - impacts science: This tag can be applied to bugs, and means that this bug causes significantly incorrect results in the model's science, at least in some configurations. Bugs that cause the model to crash are not given this label: this is reserved for problems where the model runs but gives incorrect results (which is often worse than crashing).

  • tag: enh - new science: This tag can be applied to enhancements, and means that this enhancement brings in new science capabilities. Issues with this label are generally those that are worth highlighting to scientific audiences, such as in a list of ongoing or completed enhancements presented at a CESM working group meeting. This label can also apply to primarily software engineering developments if they give new capabilities of broad scientific interest.

    • Note that some big enhancements are given their own project board. For these, we should have a small place-holder issue with this label (along with type: enhancement) so that these big enhancements show up in searches for this label.

  • tag: good first issue: This would be a good issue for someone who is just starting to contribute to CTSM.

  • tag: large: This is a large project that will take a few weeks or more.

  • tag: next: We have identified this as an issue that should get some attention soon (e.g., in the next week or two). This differs from priority: even low priority issues may be tagged next, e.g., if we want to look back and see if this is still an issue. Often an issue will be given this a label during an issue triage to note that someone should look at this and either resolve it or decide what to do about it. This can also be added to flag an issue as needing follow-up in our next weekly meeting.

    • This label should be removed once the near-term action has been taken, so that only issues that still need some near-term action keep this label. (e.g., if you look into an issue and decide it should be low priority to actually resolve it, then the tag: next label should be removed.)

    • Not all issues that are being actively worked on need to have this label: this label is meant as a convenient way to remember what we have decided to work on next, but if you don't need this label in order to remember the issue, then there's no need to apply it.

  • tag: simple bfb: This issue should be relatively easy to fix and should not change answers for any configuration (i.e., it is bit-for-bit). Therefore, it is a good candidate for being combined with other bit-for-bit issues in running the test suite. Note: it can be okay for the issue to just change a small number of diagnostic fields; in this case, whoever runs the test suite on the combined tag should confirm that only the expected fields have changed via cime/tools/cprnc/summarize_cprnc_diffs.

  • tag: support tools only: Code changes that only affect auxiliary support tools, without affecting the main CTSM code or the standard input file support chain. This would include tools in the "contrib" directory as well as other auxiliary tools that don't have standard tests and are not used as part of normal operation of either CTSM or the creation of surface datasets. For example, this includes refactor tools, tools to create lists of input files for XML, etc.