Resolving Background Job Alerts - department-of-veterans-affairs/caseflow GitHub Wiki

This page provides instruction to address problems reported by background jobs in the #appeals-job-alerts Slack channel.

Also note that there are some jobs that report to the sprint team's Slack channel, (such as StuckVirtualHearingsChecker to #appeals-tango and ReviewsWithDuplicateEpErrorChecker in #appeals-foxtrot).

StuckAppealsChecker

AppealsWithNoTasksOrAllTasksOnHoldQuery

Examine each appeal's task tree and assess if the appeal would be stuck, i.e., there are no active tasks. Historically, these problems result from Bat Team support tickets, so find the ticket and gather some understanding on what the intended state of the appeal should be. If unsure, ask for help -- don't assume.

If the root cause is from Caseflow code, let the respective sprint team know ASAP. There may be some edge case that was not considered for a recent PR.

appeals=AppealsWithNoTasksOrAllTasksOnHoldQuery.new.call
appeals.count  # this should match with the number reported in the Slack message

appeals[0].uuid # Search for a URL with this UUID in #appeals-batteam; the associate Bat Team ticket provides useful context

appeals.first.appellant # Determine if this is due to an unrecognized appellant. If so, see note below.
=> #<OtherClaimant id: 12345, ...

# Examine the task tree and determine what needs to be fixed based on the intended state of the appeal
appeals[0].treee

Note: Currently for appeals with an unrecognized appellant, we prevent the creation of dispatch tasks so the Board does not send out a letter -- Slack. Foxtrot plans to implement workflows to unblock these appeals. Do not try to fix these appeals.

Otherwise, one possible solution is to make the on_hold task to be assigned: Task.find(task_id).assigned! Another solution may be to cancel the task. Prior fixes:

Here are notes from a past meeting about an Echo PI project related to this alert.

AppealsWithClosedRootTaskOpenChildrenQuery

Ask Teams for help, for example see this Slack thread.

TasksAssignedToInactiveUsers

Check if the user is listed in Reassign Tasks for Inactive Accounts Part 3 #16275 and remind Echo to address the inactive user. If the user is not listed, let Echo know. Instructions for distributed appeals (i.e., appeals where the DistributionTask is completed) are provided in https://github.com/department-of-veterans-affairs/caseflow/issues/16275#issuecomment-847141197.

If the tasks are related to hearings (which are usually created before an appeal is distributed), let Tango know.

This Metabase table lists tasks assigned to inactive users.

OpenTasksWithClosedAtChecker

Prior investigation. Also see Clean up open tasks with closed parents #13438.

tasks=Task.open.where.not(closed_at: nil)
# Review them one at a time
tasks[0].tasks
...

# Get some stats regarding the parent tasks
cp = Task.open.joins(:parent).includes(:parent).where.not(parents_tasks: { closed_at: nil })
cp.group(:type).count

# Identify patterns (see "Prior investigation" link above) and fix them
# Document patterns on this wiki page

# Only if you're confident, fix all of them at once:
tasks_to_fix.update_all(closed_at: nil)