Metrics - department-of-veterans-affairs/caseflow GitHub Wiki

Current Monitoring and Metrics

What are Metrics Services?

In order to continuously monitor performance to help identify bottlenecks, slow processes, and areas that need optimization, we applied custom Metrics Services to Caseflow. It enables custom metrics collection, that can be tailored to the specific needs of Caseflow, making it adaptable to changing requirements over time. These allow for fine-grained data collection, enabling in-depth analysis and troubleshooting, which is crucial for diagnosing complex issues, and proactively addressing issues/errors before they impact the users.

We used the Goal-Question-Metric (GQM) approach to identify appropriate metrics for Caseflow by first identifying the information gaps we wanted to investigate. We searched the code to identify the factors that needed to be measured, such as code quality, development speed, and user satisfaction, including areas of user experience such as validation, failures, performance time, and errors that needed to be caught and defined.

Next, we identified key performance indicators (KPIs) to measure progress towards our goals. We defined the metrics that would be used to measure those KPIs and implemented them in the code to collect the data. The data is sent to Rails Logging and Datadog for analysis. We use these logs to identify areas for improvement and make changes to the code.

Why use Metrics Services?

This gives us the ability to store and view all metrics in one database which we can analyze without the use of external services or products, but these metrics can be integrated with external services or products in certain use cases.

Metric table/model

All metrics are saved to a table in our database. This table and other relevant metric dashboards can be viewed on Metabase. Ensure no PII/PHI is being saved in any metric field.

Field Name Data Type Description Example
UUID uuid Unique ID for the metric, can be used to search within various systems for the logging 30a33868-2c1b-449c-8cec-3f265a7d0e43
User ID bigint The ID of the user who generated metric. 123
Metric Name string Name of Metric caseflow.server.metric.request_latency
Metric Class string Class of metric, use reflection to find value to populate this MetricService
Metric Group string Metric group: service, etc service
Metric Message string Message or log for metric BGS:fetch veteran info fo vbms id: XXXXXXXXX
Metric Type string Type of metric: ERROR, LOG, PERFORMANCE, etc performance
Metric Product string Where in application: Queue, Hearings, Intake, VHA, etc bgs
App Name string Application name: caseflow or efolder caseflow
Metric Attributes json Store attributes relevant to the metric: OS, browser, etc {"service":"bgs", "endpoint": "claims.find_by_vbms_file_number"}
Additional Info json additional data to store for the metric {"file_number_type": "SSN"}
Sent To string Which system metric was sent to: Datadog, Rails Console, Javascript Console, etc {rails_console, dynatrace}
Sent To Info json Additional information for which system metric was sent to {"metric_group":"service","metric_name":"request_latency","metric_value":0.5868008640245534,"app_name":"caseflow","attrs":{"service":"bgs","endpoint":"claims.find_by_vbms_file_number","uuid":"30a33868-2c1b-449c-8cec-3f265a7d0e43"}}
Relevant Tables Info json Store information to tie metric to database table(s) {}
Start Time datetime When metric recording started 2023-9-5, 10:30 AM
End Time datetime When metric recording stopped 2023-9-5, 10:30 AM
Duration float Time in milliseconds from start to end 110.51
Created At datetime When the metric was created in the database 2023-9-5, 10:30 AM

How to use Metric Services

To enable the Metrics Service, the metrics_monitoring feature toggle is required for the current user.

Front End

The following frontend Metrics Services will trigger a stopwatch, recording the duration of the process automatically. The timing data and metric data to be recorded will be sent to the backend via a POST request to /metrics/v2/logs, and is stored in the metrics database table. The service will send back the return value of the function that the metric was recording.

MetricsService.js

To measure frontend metrics that involve JavaScript processes and asynchronous calls:

Use recordMetrics() when you want to record javascript proccesses, such as a looping through a large array or parsing a long string of text.

To use recordMetrics() you will need to pass three arguments:

  • The function to be recorded/tested

  • An object containing the metric data fields for the table. The required field is data, which requires a JSON format. The message field is optional, and defaults to the uuid and data of the metric. The product field is optional, but if used is required as a product type located in the Metric Model, and otherwise defaults to "caseflow".

  • The dedicated feature toggle that controls if the metrics are stored to the database. This defaults as true. Ensure it is included when calling recordMetrics() to avoid unneccesary data.

// Set up a large array
const testArray = [1,2,3...100];

// Function to loop through an array
function loopArr(arr) {
    for (let i = 0; i < arr.length; i++) {
        console.log(i)
    }
};

// Set up metric data with the fields (message, type, product, data)
const metricData = {
      message: "Testing duration of loopArr function with a large array",
      type: 'performance',
      product: 'reader',
      data: {
        "browser": "chrome",
        "OS": "Mac",
      },
    };


// Pass recordMetrics() the target function of loopArr(), metricData object, and dedicated feature toggle to enable this metric
recordMetrics(loopArr(testArray), metricData, <METRICS_FEATURE_TOGGLE>);

The use of recordAsyncMetrics() functions similarly to recordMetrics() method, but in place of a function, pass in a promise to record a metric. A metric data object in JSON format and a dedicated feature toggle are still required.

// Set the file name
const fileName = "helloworld.pdf"

// Set up a promise from a pdfJS LoadingTask
const promise = pdfjsLib.getDocument(fileName).promise;

// Set up metric data with the fields (data, type, message, product)
const metricData = {
      message: "Loading document using pdfJS",
      type: 'performance',
      product: 'reader',
      data: {
        "browser": "chrome",
        "OS": "Mac",
        "filename": `${fileName}`
      },
    };


// Pass recordAsyncMetrics() the promise, metricData object, and dedicated feature toggle
recordAsyncMetrics(promise, metricData, <METRICS_FEATURE_TOGGLE>);

ApiUtil.js

ApiUtil is used for logging successful/unsuccesful API REST calls.

To save metrics in the database while using the ApiUtil, pass in specific featureToggles in the requestOptions object:

  • metricsLogRestError records the method, the URL, and the error message
  • metricsLogRestSuccess records the method, and the URL
// Build requestOptions object with required feature toggles
let requestOptions = {
      responseType: 'application/json',
      metricsLogRestError: this.props.featureToggles.metricsLogRestError,
      metricsLogRestSuccess: this.props.featureToggles.metricsLogRestSuccess
    };

// Make API call with specified request options using ApiUtil.get 
const response = await ApiUtil.get("https://catfact.ninja/fact", requestOptions)

Back End

To enable backend Metrics Service, the metrics_monitoring is required for the current user. Using this service will trigger a stopwatch, recording the duration of the process automatically. The timing data and metric data will be saved to the metrics database table. The service will send back the return value of the function that the metric was recording.

MetricsService.record()

To use MetricsService.record() identify the method(s) for metrics collection, and wrap method(s) with MetricsService.record in a do/end block.

  • The description details the method as the metric message (defaults to a blank field). Saves as metric_message in the database table.
  • service can be used to record a specific third-party service if the method requires (defaults to "nil"). Saves as a part of metric_attributes in the database table.
  • name can be specified as the endpoint of the specific third-party service (defaults to "unknown"). Saves as a part of metric_attributes in the database table.
  • caller is the metric class, the metric params, and the current user (defaults to "nil"). This class saves to the metric_class, the user saves as the user to the database table.
# Set up a large array
array = (1..100).to_a

# Wrap service around target function passing in (description, service, name)
MetricsService.record("Printing Veteran document ID's from array",
    service: "vbms",
    name: "print_all_veteran_document_ids") do
      # Target function being measured
      array.each { |x| puts x }
end       

Rails Logging

Direct Rails Logging

Where in Code Key ID(s) Method Used Description
Idt::Api::V1::BaseController .log_error() error Rails .logger.error(error) Sends the error to the Rails logger
Idt::Api::V1::BaseController .log_error() error Rails.logger .error(error&.backtrace&.join("\n")) Handles the error along with back trace
Reader::DocumentsController .index() Appeal_id Rails .logger Some users couldn't access the documents under other profiles.
Services::metrics_service .record() info Rails .logger Logs the description after the start and end of metric services
AdminController .verify_access() info Rails.logger .info() User with specific roles couldn’t access the documents.
QueuController .verify_access() info Rails.logger .info() Redirects the user associated with case details to search.
AnnotationController .rescue_from error Rails.logger .error Validation Failed.
Idt::Api::V1::MpiController .veteran_updates() info Rails.logger .info Logs veteran info like id, deceased time and indicator
Idt::Api::V1::JobsController .create() info Rails.logger .info Displays message related to a job being pushed to queue
TeamManagementController .update() info Rails.logger .info Logs message related to existing record updation
TeamManagementController .create_judge_team() info Rails.logger .info A message with info stating that judge team is created for a specific user is displayed.
TeamManagementController .create_dvc_team() info Rails.logger .info A success message is logged when a dvc team is created for a user
TeamManagementController .create_private_bar info Rails.logger .info Used to log message when a private bar with parameters is being created
TeamManagementController .create_field_vso() info Rails.logger .info Logs a message while field vso is created.
TeamManagementController .create_national_vso() info Rails.logger .info Creating a Vso with parameters is logged.
Idt::Api::V1::BaseController .rescue_from error Rails.logger .error Logs an IDT standard error with uuid
HearingsController .rescue_from debug Rails.logger .debug Debugging message when a hearing cannot be found in VACOLS and caseflow.
ExplainAppealEventsConcern .request_issues_as_event_data() warn Rails.logger .warn Unaccounted decision issues that are left will be logged as warning.
IntakesController .log_error() error Rails.logger .error Intake error related to sentry event is displayed.
IssuesController .rescue_form ActiveRecord error Rails.logger .error A failure of Issues Controller is being popped.

Sentry/Raven

Direct Raven calls

Where in Code Key ID(s) Method Used Description
Idt::Api::V1::BaseController .log_error() error Raven .capture_exception() Handles logging errors that occur within the class
ApplicationController .rescue_from StandardError exception, error_uuid Raven .capture_exception() Logs the exception error and whether the exception was actionable
ApplicationController .on_external_error() error, error_uuid Raven .capture_exception() Logs the error and its uuid when the error is not ignorable. Used when there is a BGS::ShareError or VBMS::ClientError
ApplicationController .rescue_from Caseflow...VacolsRepositoryError exception, error_uuid Raven .capture_exception() Logs the exception and its uuid when there is a Vacols Repository Errors
AppealsController .fetch_notification_list() error, error_uuid Raven .capture_exception() Logs standard errors that occur when fetching the notification list
AppealsController .render_error() Error, appeal.type appeal.id Raven .capture_exception() Used to create a standardized error message for the controller
ClaimReviewController .render_error() error, error_uuid Raven .capture_exception() Used to create a standardized error message for the controller
DecisionReviewsController .update() Error, error_uuid Raven .capture_exception() Logs errors that occur when updating a Decision Review
IntakesController .log_error() error Raven .capture_exception() Used to create a standardized error message for the controller
IntakesController .index_props() error Raven .capture_exception() Logs standard errors that occur in the method
TasksController .update() error Raven .capture_exception() Logs errors from several custom error types while updating a Task: Caseflow::Error::InvalidEmailError AssignHearingDispositionTask::HearingAssociationMissing Caseflow::Error::VirtualHearingConversionFailed
Hearings::HearingDayController .log_error() error Raven .capture_exception() Used to create a standardized error message for the controller
Idt::AuthenticationsController .index() error Raven .capture_exception() Logs errors when encountering Caseflow::Error::InvalidOneTimeKey
Test::UsersController .show_error() error, error_uuid Raven .capture_exception() Used to create a standardized error message for the controller Note: Appears to be unused
ApplicationJob .capture_exception() error Raven .capture_exception() Used to create a standardized error message for the job if the error is not ignorable
DispatchEmailJob .send_email() error Raven .capture_exception() Logs errors encountered while sending emails
SetAppealAgeAodJob .log_error() error Raven .capture_exception() Used to create a standardized error message for the job
StatsCollectorJob .log_error() error, collector_name Raven .capture_exception() Used to create a standardized error message for the job
UpdateAppellantRepresentationJob .log_error() error Raven .capture_exception() Used to create a standardized error message for the job
UpdateCachedAppealsAttributesJob .log_error() error Raven .capture_exception() Used to create a standardized error message for the job
VANotifyStatusUpdateJob .get_current_status() error, error_uuid Raven .capture_exception() Logs errors when type of VA Notify Status type is neither an Email nor an SMS
VANotifyStatusUpdateJob .get_current_status() error, error_uuid Raven .capture_exception() Logs error when job encounters a Caseflow::Error::VANotifyApiError
WarmBgsCachesJob .warm_people_caches() error Raven .capture_exception() Logs standard errors encountered during the function
WarmBgsCachesJob .warm_participant_caches() error Raven .capture_exception() Logs standard errors encountered during the function
WarmBgsCachesJob .warm_ro_participant_caches() error Raven .capture_exception() Logs standard errors encountered during the function
WarmBgsCachesJob .warm_veteran_attribute_caches() error Raven .capture_exception() Logs standard errors encountered during the function
Hearings::GeomatchAndCacheAppealJob retry_on() exception Raven .capture_exception() Logs an error when the job has retried 10 times
Hearings::SendEmail .send_email() error Raven .capture_exception() Logs standard and custom errors encountered while sending an email
Hearings::SendEmail .create_sent_hearing_email_event() error Raven .capture_exception() Logs standard errors encountered while creating the sent hearing email event
Hearings::SendSentStatusEmail .call() error Raven .capture_exception() Logs standard and custom errors encountered while sending an email
VirtualHearings::CreateConferenceJob retry_on() exception Raven .capture_exception() Logs an error when the job has encountered the PexipApiError 10 times
VirtualHearings::CreateConferenceJob .send_emails() exception Raven .capture_exception() Logs standard errors encountered while sending an email
VirtualHearings::CreateConferenceJob .generate_links_and_pins() exception Raven .capture_exception() Logs standard errors encountered while generating links and pins for virtual hearings
VirtualHearings::DeleteConferencesJob retry_on() exception Raven .capture_exception() Logs an error when the job has encountered the DeleteConferencesJobFailure error 5 times
DecisionReview .ratings_with_issues_or_decisions() error Raven .capture_exception() Logs an error when the model encounters the PromulgatedRating::LockedRatingError
DecisionReviewIntake . after_validated_pre_start!() error Raven .capture_exception() Logs an error when the model is running during start and encounters the error: EndProductEstablishment::EstablishedEndProductNotFound
DistributedCase .flag_redistribution() Case id, task.appeal.uuid Raven.capture_message() Logs a message when a distributed case already exists for the same appeal
EstablishClaim .try_prepare_with_decision!() error, claim id Raven .capture_exception() Logs an error when the model encounters a VBMS::ClientError when trying to prepare a decision
HearingDay .log_error() error, hearing_day.id Raven .capture_exception() Used to create a standardized error message for the model
LegacyHearing .vacols_hearing_exists?() error Raven .capture_exception() Logs an error when the model encounters the VacolsRecordNotFound error
PromulgatedRating .retry_fetching_rating_profile() error Raven .capture_exception() Logs an error when retrying a fetch and encountering the errors BGS::ShareError Rating::NilRatingProfileListError
RampRefiling .create_contentions_on_new_end_product!() error Raven .capture_exception() Logs standard errors encountered by the model while performing this task
Veteran .handle_bgs_share_error() error Raven .capture_exception() Logs an error unless the error message is a Sensitive File
AppealConcern .timezone_identifier_for_address() error Raven .capture_exception() Logs a standard error encountered by the model while trying to identify the time zone for an address
AssociatedVacolsModel .check_and_load_vacols_data!() LazyLoadingDisabledError Raven .capture_exception() Logs an error encountered while trying to load vacols data
HearingConcern .rescue_and_check_toggle_veteran_date_of_death_info() error Raven .capture_exception() Logs standard and custom errors encountered by the model while checking the veteran date of death info
ConferenceLink .generate_links_and_pins() error Raven .capture_exception() Logs custom errors while the model is trying to generate links and pins: VirtualHearings::LinkService::PINKeyMissingError VirtualHearings::LinkService::URLHostMissingError VirtualHearings::LinkService::URLPathMissingError
SentHearingEmailEvent .invalid_reported_status_failure() reported_status Raven .capture_exception() Logs an error message when an invalid reported status is entered
SentHearingEmailEvent .email_already_sent_failure() external_message_id Raven .capture_exception() Logs an error message when a previous email has failed to be delivered to the email addesss provided
WorkQueue::TaskColumnSerializer :veteran_appellant_deceased error Raven .capture_exception() Logs an error when the model encounters the BGS::PowerOfAttorneyFolderDenied error
InformalHearingPresentationTask .update_to_new_poa() error Raven .capture_exception() Log standard errors encountered while updating power of attorney
ScheduleHearingTask .cancel_parent_task() parent.id Raven .capture_exception() Logs an error when trying to cancel a parent HearingTask that still has active child tasks
HearingRepository .hearings_for() Error, vacols_record.hearing_pkseq Raven .capture_exception() Logs an error when a regional office for a case hearing is not found (RegionalOffice::NotFoundError)
DocumentFetcher .warn_about_same_vbms_document_id() error_message Raven .capture_exception() Logs a DuplicateVbmsDocumentIdError
RedistributedCase .alert_case_not_found() error, case_id Raven .capture_exception() Logs an error when a case is not found
RedistributedCase .alert_existing_distributed_case_not_unique() error, case_id, judge.css_id Raven .capture_exception() Logs an error when another DistributedCase already exists
VaDotGovAddressValidator .remove_missing_facilities() cacility missing_ids, error Raven .capture_exception() Logs an error when given facility ids are not found
VirtualHearings::ResendVirtualHearingEmailsService .reset_sent_status_and_send() error Raven .capture_exception() Logs standard and custom errors encountered during the this service's method
VirtualHearings::ResendVirtualHearingEmailsService . should_resend_email?() error Raven .capture_exception() Logs standard and custom errors encountered while determining if an email should be resent
LegacyAppeal .assigned_to_location() Legacy appeal external_id Raven.capture_message() Logs a message when the Legacy Appeal only has closed tasks left
LegacyAppeal .assigned_to_location() Legacy appeal external_id Raven.capture_message() Logs a message when the Legacy Appeal is open but has no tasks
Task .put_on_hold_due_to_new_child_task() Task id Raven.capture_message() Logs a message when cloning a task to create a child task
JudgeAssignTask .begin_decision_review_phase() Message, time Raven.capture_message() Logs a message that notes the time and that the type from Assign to DecisionReview is still being changed
RootTask .when_child_task_created() child_task.id, root_task.id Raven.capture_message() Logs a message that a child task has been created from a root task but didn’t update RootTask status
HearingRepository .hearings_for() vacols_ids Raven.capture_message() Logs a message that a hearings_for has been sent for a non-unique vacols id
InitialTasksFactory .warn_poa_not_a_representative() poa_participant_id, appeal.id Raven.capture_message() Logs a message when POA is not a representative
HearingBadge.jsx componentDidMount() error Raven.captureException() Logs and displays errors which occur while getting hearings for an appeal

Indirect Raven/Sentry calls

Where in Code Key ID(s) Method Used Description
Idt::Api::V1::BaseController rescue_from error BaseController .log_error() Used multiple times to ensure errors are logged, including custom error types
AppealsController .update_power_of_attorney() error AppealsController .render_error() Logs standard errors that occur when updating the power of attorney
ClaimReviewController .edit() error ClaimReviewController .render_error() Logs standard errors that occur when editing a claim review
IntakesController .create() error IntakesController .log_error() Logs standard errors that occur when creating Intakes
IntakesController .review() error IntakesController .log_error() Logs standard errors that occur when rendering an Intake review
IntakesController .complete() error IntakesController .log_error() Logs standard errors what occur when marking an Intake as complete
Hearings::HearingDayController .show() error Hearings::HearingDayController .log_error() Logs errors from several custom error types while displaying a Hearing Day: PINKeyMissingError, URLHostMissingError, URLPathMissingError
ApplicationJob rescue_from error ApplicationJob .capture_exception() Logs error when encountering custom errors during the execution of the job: Caseflow::Error::TransientError, VBMS::ClientError, BGS::ShareError
SetAppealAgeAodJob .perform() error SetAppealAgeAodJob .log_error() Logs errors encountered during the execution of the job
StatsCollectorJob .perform() error StatsCollectorJob .log_error() Logs standard errors encountered while executing the job
StatsCollectorJob .run_collectors() error StatsCollectorJob .log_error() Logs standard errors encountered while starting up the collectors
UpdateAppellantRepresentationJob .perform() error UpdateAppellantRepresentationJob() Logs standard errors encountered while executing the job
UpdateCachedAppealsAttributesJob .perform() error UpdateCachedAppealsAttributesJob .log_error() Logs standard error encountered while executing the job
HearingDay .generate_link_on_create() error HearingDay .log_error() Logs standard errors encountered while the model is creating a link during creation

Datadog

Direct Datadog Calls

Where in Code Key ID(s) Method Used Description
DataDogService .increment_counter() stat_name Datadog::Statsd .increment() Updates a datadog counter
DataDogService .emit_gauge() stat_name Datadog::Statsd .gauge() Sends given statistics to datadog
DataDogService .histogram() stat_name Datadog::Statsd .histogram() Creates a datadog histogram

DataDogService Calls

Where in Code Key ID(s) Method Used Description
MetricsService .record() app, service DataDogService .emit_gauge() Records the time it takes for a given service to run
MetricsService .increment_datadog_counter() Metric_name, service, endpoint_name DataDogService .increment_counter() Increments the count of the given counter
Config.ru emit_datadog_point() threads DataDogService .emit_gauge() Records count of puma threads
ApplicationController .handle_non_critical_error() endpoint, error_type, err.code DataDogService .increment_counter() Keeps count of errors based on error type and code
CollectDataDogMetrics .emit_datadog_point() Db_name, connection type DataDogService .emit_gauge() Standardizes datadog message for tracking database connection counts
Metrics::V1::HistogramController .create() metric DataDogService .histogram() Creates a datadog histogram based on a given metric
CaseflowJob .datadog_report_runtime() metric_group_name DataDogService .record_runtime() Records run time for caseflow jobs
CaseflowJob .datadog_report_time_segment() segment DataDogService .emit_gauge() Records run time for a given caseflow job segment
DispatchEmailJob .external_message_id() “Email.sent”, email type DataDogService .increment_counter() Keeps count of how many emails are sent
UpdateAppellantRepresentationJob .increment_task_count() METRIC_GROUP_NAME, task_effect, appeal_id DataDogService .increment_counter() Keeps track of tasks done for an appeal
UpdateCachedAppealsAttributesJob .increment_vacols_update_count() APP_NAME, METRIC_GROUP_NAME DataDogService .increment_counter() Keeps track of vacols update counts
UpdateCachedAppealsAttributesJob .increment_appeal_count() APP_NAME, METRIC_GROUP_NAME DataDogService .increment_counter() Keeps track of appeal counts
UpdateCachedAppealsAttributesJob .record_success_in_datadog() APP_NAME, METRIC_GROUP_NAME DataDogService .increment_counter() Keeps track of successful runs of the job
UpdateCachedAppealsAttributesJob .record_error_in_datadog() APP_NAME, METRIC_GROUP_NAME DataDogService .increment_counter() Keeps track of unsuccessful runs of the job
WarmBgsCachesJob .warm_poa_for_oldest_cached_records() Hearings::SendEmail.external_message_id(), DATADOG_METRICS.HEARINGS.APP_NAME, ATADOG_METRICS.HEARINGS.VIRTUAL_HEARINGS_GROUP_NAME DataDogService .increment_counter() Keeps track of number of emails sent
Hearings::SendSentStatusEmail .log_to_datadog() message, hearing.id, hearing.class.name DataDogService .increment_counter() Increments a counter to keep track results of sending emails
Memberships::SendMembershipRequestMailerJob .external_message_id() “Email.error”, requestor, requests DataDogService .increment_counter() Increments error count for failed emails
VirtualHearings::CreateConferenceJob .create_conference() "Created_conference.failed”, hearing_id DataDogService .increment_counter() Keeps track of failed jobs for creating a virtual hearing conference
VirtualHearings::CreateConferenceJob .create_conference() "Created_conference.successful", hearing_id DataDogService .increment_counter() Keeps track of successful jobs for creating a virtual hearing conference
VirtualHearings::DeleteConferencesJob .count_deleted_and_log() deleted_conferences.successful DataDogService .increment_counter() Keeps track of successful jobs for deleting a virtual hearing conference
VirtualHearings::DeleteConferencesJob .count_deleted_and_log() deleted_conferences.failed DataDogService .increment_counter() Keeps track of failed jobs for deleting a virtual hearing conference
LegacyAppeal .veteran_file_number() database_disagreement, external_id DataDogService .increment_counter() Keeps track of number of disagreements between vacols and caseflow file numbers
AppealConcern .timezone_identifier_for_address() error.country_code DataDogService .increment_counter() Keeps track of Ambiguous Time zone Error count
BaseHearingUpdateForm .create_or_update_virtual_hearing() “cancelled_virtual_hearing.successful” DataDogService .increment_counter() Keeps track of number of successfully canceled virtual hearings
BaseHearingUpdateForm .create_or_update_virtual_hearing() “updated_virtual_hearing.successful” DataDogService .increment_counter() Keeps track of number of successfully updated virtual hearings
BaseHearingUpdateForm .create_or_update_virtual_hearing() “created_virtual_hearing.successful” DataDogService .increment_counter() Keeps track of number of successfully created virtual hearings
BusinessMetrics .record() app_name, service DataDogService .increment_counter() Used to keep track of the count of a given service
DataDogService .record_runtime() app_name, metric_group, metric_name DataDogService .emit_gauge() Formats information to send to datadog
GeomatchService .record_geomatched_appeal() appeal.external_id DataDogService .increment_counter() Keeps track of number of geomatched appeals
Hearings::ReminderService .send_to_datadog() Type, hearing.id DataDogService .increment_counter() Keeps track of how many reminders of a given type have been created

MetricsService Datadog Calls

Where in Code Key ID(s) Method Used Description
MetricsService .record() service MetricsService. increment_datadog_counter() Increments the counter for request_errors for a service
MetricsService .record() service MetricsService. increment_datadog_counter() Increments the counter for request_attempts for a service
AppealsController .show() appeal_id MetricsService. record() Record time it takes to load or create an appeal
LegacyTasksController .index() user_id MetricsService. record() Record time it takes to load all appeal tasks for a user
Idt::Api::V2::AppealsController .reader_appeal() appeal_id MetricsService. record() Record time it takes to load appeal data
Idt::Api::V2::AppealsController .appeal_documents() appeal_id MetricsService. record() Record time it takes to load appeal document data
Reader::AppealController .show() vacols_id MetricsService. record() Record time it takes to get appeal info related to a Vacols id
Reader::DocumentsController .index() appeal_id MetricsService. record() Record time it takes to get document data for an appeal
FetchAllActiveAmaAppealsJob .add_record_to_appeal_states_table() appeal_id MetricsService. record() Record time it takes to update a record in the Appeal States Table
FetchAllActiveAmaAppealsJob .add_record_to_appeal_states_table() appeal_id MetricsService. record() Record time it takes to create a record in the Appeal States Table
FetchAllActiveLegacyAppealsJob .add_record_to_appeal_states_table() appeal_id MetricsService. record() Record time it takes to update a record in the Appeal States Table
FetchAllActiveLegacyAppealsJob .add_record_to_appeal_states_table() appeal_id MetricsService. record() Record time it takes to create a record in the Appeal States Table
QuarterlyNotificationsJob .perform() appeal_id MetricsService. record() Record time it takes to create a Quarterly Notification
AttorneyCaseReview .update_in_vacols!() task_id MetricsService. record() Record time it takes to reassign a case
JudgeCaseAssignmentToAttorney .assign_to_attorney!() vacols_id MetricsService. record() Record time it takes to assign a case to an attorney
JudgeCaseAssignmentToAttorney .reassign_to_attorney() vacols_id MetricsService. record() Record time it takes to reassign a case to an attorney
JudgeCaseReview .update_in_vacols!() task_id MetricsService. record() Record time it takes to update a Judge Case Review
AppealCancelled .update_appeal_state_when_appeal_cancelled() appeal.id MetricsService. record() Record time it takes to set an appeal’s canceled state to true
AppealDocketed .create_tasks_on_intake_success!() Appeal id MetricsService. record() Record time it takes to send a notification to VA notify
AppealDocketed .docket_appeal() appeal.id MetricsService. record() Record time it takes to send a notification to VA notify
AppealDocketed .update_appeal_state_when_appeal_docketed() appeal.id MetricsService. record() Record time it takes to set an appeal’s docketed status to true
HearingPostponed .update_appeal_states_on_hearing_postponed() appeal.id MetricsService. record() Record time it takes to update an appeal’s HEARING_POSTPONED status
HearingPostponed .update_appeal_states_on_hearing_postponed() appeal.id MetricsService. record() Record time it takes to update an appeal’s HEARING_POSTPONED status
HearingScheduled .update_appeal_states_on_hearing_scheduled() appeal.id MetricsService. record() Record time it takes to update an appeal’s HEARING_SCHEDULED status
HearingScheduledInError .update_appeal_states_on_hearing_scheduled_in_error() appeal.id MetricsService. record() Record time it takes to update an appeal’s SCHEDULED_IN_ERROR status
HearingScheduledInError .update_appeal_states_on_hearing_scheduled_in_error() appeal.id MetricsService. record() Record time it takes to update an appeal’s SCHEDULED_IN_ERROR status
HearingWithdrawn .update_appeal_states_on_hearing_withdrawn() appeal.id MetricsService. record() Record time it takes to update an appeal’s HEARING_WITHDRAWN status
HearingWithdrawn .update_appeal_states_on_hearing_withdrawn() appeal.id MetricsService. record() Record time it takes to update an appeal’s HEARING_WITHDRAWN status
IhpTaskCancelled .update_appeal_state_when_ihp_cancelled() appeal.id MetricsService. record() Record time it takes to update an appeal’s VSO_IHP statuses to false
IhpTaskComplete .update_from_params() appeal.id MetricsService. record() Record time it takes to send a notification to VA Notify
IhpTaskComplete .update_appeal_state_when_ihp_completed() appeal.id MetricsService. record() Record time it takes to update an appeal’s VSO_IHP statuses
IhpTaskPending .create_ihp_tasks!() parent.appeal.id MetricsService. record() Record time it takes to send a notification to VA Notify
IhpTaskPending .create_from_params() appeal.id MetricsService. record() Record time it takes to send a notification to VA Notify
IhpTaskPending .update_appeal_state_when_ihp_created() appeal.id MetricsService. record() Record time it takes to update an appeal’s VSO_IHP statuses
PrivacyActCancelled .update_appeal_state_when_privacy_act_cancelled() appeal.id MetricsService. record() Record time it takes to update an appeal’s PRIVACY_ACT_PENDING status to false
PrivacyActComplete .update_appeal_state_when_privacy_act_complete() appeal.id MetricsService. record() Record time it takes to update an appeal’s PRIVACY_ACT statuses
PrivacyActPending .update_appeal_state_when_privacy_act_created() appeal.id MetricsService. record() Record time it takes to update an appeal’s PRIVACY_ACT statuses
VACOLS::Case .aod() vacols_ids MetricsService. record() Record time it takes to query cases that match the criteria
VACOLS::Case .remand_return_date() vacols_ids MetricsService. record() Record time it takes to query cases that match the criteria
VACOLS::Case .previous_active_location() bfkey MetricsService. record() Record time it takes to query cases that match the criteria
VACOLS::CaseAssignment .exists_for_appeals() vacols_ids MetricsService. record() Record time it takes to query case assignments
VACOLS::CaseHearing .create_hearing!() “create_hearing!” MetricsService. record() Record time it takes to create a hearing
VACOLS::CaseHearing .update_hearing!() hearing_pkseq MetricsService. record() Record time it takes to update a hearing
VACOLS::CaseIssue .descriptions() vacols_ids MetricsService. record() Record time it takes to query case issue descriptions
VACOLS::Correspondent .update_veteran_nod() Veteran_ssn, veteran_pat MetricsService. record() Record time it takes to update a veteran’s NOD
VACOLS::Note .generate_primary_key() bfkey MetricsService. record() Record time it takes to generate the primary key for a note
VACOLS::Note .create!() note[case_id] MetricsService. record() Record time it takes to create a note
VACOLS::Note .delete!() note[case_id] MetricsService. record() Record time it takes to delete a note
VACOLS::Representative .update_vacols_rep_type!() case_id MetricsService. record() Record time it takes to update the rep type for a case
VACOLS::Representative .update_vacols_rep_table!() bfkey MetricsService. record() Record time it takes to update the rep name
VacolsLocationBatchUpdater .call() “batch_update_vacols_location” MetricsService. record() Record time it takes to do a batch update on records in tables
AppealRepository .vacols_records_for_appeals() “eager_load_legacy_appeals_batch” MetricsService. record() Record time it takes to load legacy appeals
AppealRepository .load_vacols_data() appeal.vacols_id MetricsService. record() Record time it takes to load case records
AppealRepository .appeals_by_vbms_id() “appeals_by_vbms_id” MetricsService. record() Record time it takes to load cases that match the query
AppealRepository .appeals_by_vbms_id_with_preloaded_status_api_attrs() “appeals_by_vbms_id_with_preloaded_status_api_attrs” MetricsService. record() Record time it takes to load cases that match the query
AppealRepository .appeals_ready_for_hearing() “appeals_ready_for_hearing” MetricsService. record() Record time it takes to load cases that match the query
AppealRepository .load_vacols_data_by_vbms_id() appeal.vbms_id MetricsService. record() Record time it takes to load cases that match the query
AppealRepository .remands_ready_for_claims_establishment() “remands_ready_for_claims_establishment” MetricsService. record() Record time it takes to load cases that match the query
AppealRepository .amc_full_grants() outcoded_after MetricsService. record() Record time it takes to load cases that match the query
AppealRepository .certify() appeal.vacols_id MetricsService. record() Record time it takes to verify and save a case record
AppealRepository .load_user_case_assignments_from_vacols() css_id MetricsService. record() Record time it takes to load active cases for the user
AppealRepository .docket_counts_by_priority_and_readiness() “docket_counts_by_priority_and_readiness” MetricsService. record() Record time it takes to return the CaseDocket attribute
AppealRepository .genpop_priority_count() “genpop_priority_count” MetricsService. record() Record time it takes to return the CaseDocket attribute
AppealRepository .not_genpop_priority_count() "not_genpop_priority_count" MetricsService. record() Record time it takes to return the CaseDocket attribute
AppealRepository .priority_ready_appeal_vacols_ids() "priority_ready_appeal_vacols_ids" MetricsService. record() Record time it takes to return the CaseDocket attribute
AppealRepository .nod_count() "nod_count" MetricsService. record() Record time it takes to return the CaseDocket attribute
AppealRepository .regular_non_aod_docket_count() "regular_non_aod_docket_count" MetricsService. record() Record time it takes to return the CaseDocket attribute
AppealRepository .latest_docket_month() "latest_docket_month" MetricsService. record() Record time it takes to return the CaseDocket attribute
AppealRepository .docket_counts_by_month() "docket_counts_by_month" MetricsService. record() Record time it takes to return the CaseDocket attribute
AppealRepository .age_of_n_oldest_genpop_priority_appeals() "age_of_n_oldest_genpop_priority_appeals" MetricsService. record() Record time it takes to return the CaseDocket attribute
AppealRepository .age_of_n_oldest_nonpriority_appeals_available_to_judge() "age_of_n_oldest_nonpriority_appeals_available_to_judge" MetricsService. record() Record time it takes to return the CaseDocket attribute
AppealRepository .age_of_oldest_priority_appeal() "age_of_oldest_priority_appeal" MetricsService. record() Record time it takes to return the CaseDocket attribute
AppealRepository .age_of_oldest_priority_appeal_by_docket_date() "age_of_oldest_priority_appeal" MetricsService. record() Record time it takes to return the CaseDocket attribute
AppealRepository .age_of_n_oldest_priority_appeals_available_to_judge() "age_of_n_oldest_priority_appeals_available_to_judge" MetricsService. record() Record time it takes to return the CaseDocket attribute
AppealRepository .nonpriority_decisions_per_year() "nonpriority_decisions_per_year" MetricsService. record() Record time it takes to return the CaseDocket attribute
AppealRepository .distribute_priority_appeals() "distribute_priority_appeals" MetricsService. record() Record time it takes to return the CaseDocket attribute
AppealRepository .distribute_nonpriority_appeals() "distribute_nonpriority_appeals" MetricsService. record() Record time it takes to return the CaseDocket attribute
HearingRepository .load_vacols_data() hearing.vacols_id MetricsService. record() Record time it takes to load a case hearing
IssueRepository .create_vacols_issue!() vacols_id MetricsService. record() Record time it takes to create a vacols issue
IssueRepository .update_vacols_issue!() vacols_id, vacols_sequence_id MetricsService. record() Record time it takes to update a vacols issue
IssueRepository .delete_vacols_issue!() vacols_id, vacols_sequence_id MetricsService. record() Record time it takes to delete a vacols issue
PowerOfAttorneyRepository .load_vacols_data() poa.vacols_id MetricsService. record() Record time it takes to load vacols data for a POA
QueueRepository .tasks_for_user() “fetch user tasks” MetricsService. record() Record time it takes to fetch tasks for a user
QueueRepository .tasks_for_appeal() “fetch appeal tasks” MetricsService. record() Record time it takes to fetch tasks for an appeal
QueueRepository .appeals_by_vacols_ids() “fetch appeals and associated info for tasks” MetricsService. record() Record time it takes to fetch appeals and related info
AppealFinder .find_appeals_with_file_numbers() file_numbers MetricsService. record() Record time it takes to fetch appeal info for the given file numbers
AppealFinder .find_appeals_for_vso_user() “Get vso appeals information for veterans” MetricsService. record() Record time it takes to fetch appeals for a given participant
ExternalApi::BgsAddressFinder .fetch_addresses() participant_id MetricsService. record() Record time it takes to get the address for a participant
ExternalApi::BGSService .get_end_products() vbms_id MetricsService. record() Records time it takes get end products for a given vbms
ExternalApi::BGSService .cancel_end_product() veteran_file_number, end_product_code, end_product_modifier, payee_code, benefit_type MetricsService. record() Records time it takes to cancel a given end product
ExternalApi::BGSService .update_benefit_claim() veteran_file_number, payee_code, claim_date, benefit_type_code, modifier, new_code MetricsService. record() Records time it takes to update a benefit claim
ExternalApi::BGSService .fetch_veteran_info() vbms_id MetricsService. record() Records time it takes to fetch a veteran’s info based on a given vbms
ExternalApi::BGSService .fetch_person_info() participant_id MetricsService. record() Records time it takes to fetch a person’s info based on a given participant id
ExternalApi::BGSService .fetch_person_by_ssn() ssn MetricsService. record() Records time it takes to fetch a person’s info based on a given ssn
ExternalApi::BGSService .fetch_poa_by_file_number_by_claimants() file_number MetricsService. record() Records time it takes to fetch the poa for a file
ExternalApi::BGSService .fetch_poa_by_file_number_by_org() file_number MetricsService. record() Records time it takes to fetch the poa for a file
ExternalApi::BGSService .fetch_poas_by_participant_id() participant_id MetricsService. record() Records time it takes to fetch the poas that a participant holds
ExternalApi::BGSService .fetch_poas_by_participant_ids() participant_ids MetricsService. record() Records time it takes to fetch the poas that a group of participant hold
ExternalApi::BGSService .fetch_limited_poas_by_claim_ids() claim_ids MetricsService. record() Records time it takes to fetch the poas associated with a a set of claims
ExternalApi::BGSService .fetch_poas_list() “fetch list of poas” MetricsService. record() Records time it takes to fetch all powers of attorney
ExternalApi::BGSService .get_security_profile() “security profile” MetricsService. record() Records time it takes to get the security profile of a user and/or station
ExternalApi::BGSService .can_access?() vbms_id MetricsService. record() Records time it takes to determine if the user can access a file
ExternalApi::BGSService .fetch_ratings_in_range() participant_id MetricsService. record() Records time it takes to get all ratings of a participant within a given date range
ExternalApi::BGSService .fetch_rating_profile() participant_id MetricsService. record() Records time it takes to get a participant’s rating profile on a given date
ExternalApi::BGSService .fetch_rating_profiles_in_range() participant_id MetricsService. record() Records time it takes to get a participant’s rating profiles within a given date range
ExternalApi::BGSService .fetch_claimant_info_by_participant_id() participant_id MetricsService. record() Records time it takes to fetch claimant info
ExternalApi::BGSService .find_all_relationships() participant_id MetricsService. record() Records time it takes to fetch relationships for a given participant
ExternalApi::BGSService .get_participant_id_for_css_id_and_station_id() css_id, station_id MetricsService. record() Records time it takes to find the participant id for a user
ExternalApi::BGSService .find_claimant_letters() document_id MetricsService. record() Records time it takes to fetch the claimant letter for a given document
ExternalApi::BGSService .manage_claimant_letter_v2!() program_type_cd, claimant_participant_id MetricsService. record() Records time it takes to create a claimant letter
ExternalApi::BGSService .generate_tracked_items!() claim_id MetricsService. record() Records time it takes to generate tracked items for a claim
ExternalApi::BGSService .find_contentions_by_claim_id() claim_id MetricsService. record() Records time it takes to find a veteran’s contentions for a claim
ExternalApi::BGSService .find_current_rating_profile_by_ptcpnt_id() participant_id MetricsService. record() Records time it takes to get the current rating profile of a participant
ExternalApi::BGSService .pay_grade_list() “fetch list of pay grades” MetricsService. record() Records time it takes to fetch the pay grade list
ExternalApi::BgsVeteranStationUserConflict .employee_dtos() veteran_participant_id MetricsService. record() Records time it takes to fetch a dtos employee by participant id
ExternalApi::BgsVeteranStationUserConflict .sensitivity_level() veteran_participant_id MetricsService. record() Records time it takes to fetch the sensitivity level by participant id
ExternalApi::EfolderService .send_efolder_request() url MetricsService. record() Records time it takes to resolve a GET request to eFolder
ExternalApi::GovDeliveryService .send_gov_delivery_request() url, method.to_s.upcase MetricsService. record() Records time it takes to send a gov delivery request
ExternalApi::MPIService .search_people_info() last_name, first_name, middle_name, ssn, date_of_birth, gender, address, telephone MetricsService. record() Records time it takes to find a person based on the given criteria
ExternalApi::MPIService .fetch_person_info() icn MetricsService. record() Records time it takes to find a person’s info based on an icn
ExternalApi::PexipService .send_pexip_request() host, method.to_s.upcase, url MetricsService. record() Records the time it takes to send a pexip request
ExternalApi::VADotGovService .send_va_dot_gov_request() method.to_s.upcase, url MetricsService. record() Records time it takes to send a VA.gov request
ExternalApi::VANotifyService .send_va_notify_request() method.to_s.upcase, url MetricsService. record() Records time it takes to send a VA notify request
ExternalApi::VBMSRequest .call() request.class, request.id MetricsService. record() Records time it takes to send a VBMS request
VBMSCaseflowLogger .send_and_log_request() vbms_id MetricsService. record() Records time it takes to send a VBMS request
VBMSCaseflowLogger .call_and_log_service() Vbms_id, service.class MetricsService. record() Records time it takes to call a VBMS service
config/initializers/warmup_vacols.rb initial_pool_size MetricsService. record() Records the time it takes to warm up the vacols connections

CaseflowJob Datadog Calls

Where in Code Key ID(s) Method Used Description
DataIntegrityChecksJob .perform() Segment klass.underscore CaseflowJob. datadog_report_time_segment() Records runtime for each data integrity Checker that is run
ETLBuilderJob .sweep_etl()l “etl_sweeper” CaseflowJob. datadog_report_time_segment() Records runtimes of ETL sweeper jobs
ETLBuilderJob .build_etl() “etl_builder” CaseflowJob. datadog_report_time_segment() Records runtimes of ETL builder jobs
NightlySyncsJob .sync_vacols_users() "sync_users_from_vacols" CaseflowJob. datadog_report_time_segment() Records runtimes of Vacols user sync jobs
NightlySyncsJob .sync_vacols_cases() “sync_cases_from_vacols” CaseflowJob. datadog_report_time_segment() Records runtimes of Vacols case sync jobs
NightlySyncsJob .sync_bgs_attorneys() “sync_bgs_attorneys” CaseflowJob. datadog_report_time_segment() Records runtimes of BGS attorney sync jobs
PushPriorityAppealsToJudgesJob .perform() "priority_appeal_push_job" CaseflowJob. datadog_report_time_segment() Records runtimes of pushing priority appeals to judges
UpdateAppellantRepresentationJob .log_error() METRIC_GROUP_NAME CaseflowJob. datadog_report_runtime() Records runtime of the metric group when an error occurs
UpdateCachedAppealsAttributesJob .perform() “Cache_ama_appeals”, ama_appeals_start CaseflowJob. datadog_report_time_segment( Records runtime for ama appeals caching jobs
UpdateCachedAppealsAttributesJob .perform() “cache_legacy_appeals”, legacy_appeals_start CaseflowJob. datadog_report_time_segment( Records runtime for legacy appeals caching
UpdateCachedAppealsAttributesJob .cache_legacy_appeals() “cache_legacy_appeal_postgres_data”, cache_postgres_data_start CaseflowJob. datadog_report_time_segment() Records
UpdateCachedAppealsAttributesJob .cache_legacy_appeals() “cache_legacy_appeal_vacols_data”, cache_vacols_data_start CaseflowJob. datadog_report_time_segment() Records
UpdateCachedAppealsAttributesJob .log_error() METRIC_GROUP_NAME CaseflowJob. datadog_report_runtime() Records runtime of the job

Other Indirect Datadog Calls

Where in Code Key ID(s) Method Used Description
CollectDataDogMetrics .collect_postgres_metrics() "postgres", status count CollectDataDogMetrics .emit_datadog_point() Keeps count of active/dead/idle postgres connections
UpdateAppellantRepresentationJob .perform() appeal.id UpdateAppellantRepresentationJob .increment_task_count() Keeps track of new/closed/error task counts for appeals
UpdateCachedAppealsAttributesJob .perform() n/a UpdateCachedAppealsAttributesJob .record_success_in_datadog() Increments successful job count
UpdateCachedAppealsAttributesJob .log_error() n/a UpdateCachedAppealsAttributesJob .record_error_in_datadog() Keeps track of number of errors the job encounters
Hearings::SendSentStatusEmail .log() Email message Hearings::SendSentStatusEmail .log_to_datadog() Passes through the email message to datadog incrementer
Hearings::SendSentStatusEmail .call() “sent admin email” Hearings::SendSentStatusEmail .log() Increments sent email count
Hearings::SendSentStatusEmail .send_email() “failure to send email” Hearings::SendSentStatusEmail .log() Increments sent email failure count
Hearings::SendSentStatusEmail .email_missing?() “email missing” Hearings::SendSentStatusEmail .log() Increments missing email count