VRO RabbitMQ Strategy - department-of-veterans-affairs/abd-vro GitHub Wiki

Scope

The goal of this page is to identify the strategy that VRO implements for Partner Teams' integrations with VRO microservices via RabbitMQ. This document includes:

  • VRO & Partner Team Responsibilities
    • Exchange / Queue Declarations and Bindings
  • Global Exchange / Queue Policies
  • Requests / Response Model
  • Stream / Push Model
  • Partner Team Recommendations

For more information on RabbitMQ specifics or AMQP protocol concepts, consult the documentation on the RabbitMQ website.

VRO Microservice Responsibilities

  • declaring exchanges and their type (direct exchange, fanout exchange, topic exchange, or headers exchange)
  • declaring request queues
  • binding request queues to the exchanges
  • setting all global exchange / queue policies (Dead-letter exchanges, TTL, Queue Length Limit, etc.)
  • setting any VRO's application queue/exchange parameters that differ from VRO global policies via declarations

The following settings are expected for exchange declarations:

  • must be declared durable (exchange will survive a broker restart)
  • must be auto-deleted (exchange is deleted when the last queue is unbound from it)

The following settings are expected for request queue declarations:

  • must be declared durable (queue will survive a broker restart)
  • must be auto-deleted (queue that has had at least one consumer will be deleted automatically when the last consumer has unsubscribed (disconnected))
  • must not be exclusive (queue is able to have more than a single connection besides declaring service)

Partner Team Responsibilities

In order for partner teams to communicate via RabbitMQ to VRO's microservices, queues and exchanges must be declared identically to the VRO microservice that will be processing requests and supplying responses. Failure to do so will result in the applications' inability to consume or publish to an exchange/queue, and depending on the RabbitMQ library client, a runtime exception.

The following settings are expected for exchange declarations:

  • must match VRO declarations for exchanges
  • must not publish messages to the default exchange

The following settings are expected for request queue declarations:

  • must match VRO declarations for queues
  • setting any application response queue parameters that differ from VRO global policies via declarations

The following settings are expected for response queue declarations:

  • must declare response queues as necessary
  • must bind response queues to the appropriate exchanges

Publishing request requirements:

  • must provide reply_to property in request message
  • must correlate requests made to responses received by using correlation_id property in request message
    • must also keep track of those correlation_ids
  • must implement client time-out or retries policies for requests where expected response was never received
  • must provide any other message level properties such as delivery_mode, see here for more info on message level properties in request message
  • must implement any sort of publisher acknowledgements if desired

Message consumption

  • must provide manual message consumer acknowledgements, rejections, or negative acknowledgements to the server upon receiving a response message
    • must validate the correlation_id if consuming a message as a response to a request

Global Exchange / Queue Policies

TBD See policies. Consider

Request / Response Model

In order to allow multiple partner teams to utilize downstream VRO microservices, the request and response model for each of the VRO's microservices through RabbitMQ shall be consistent. Keep in mind, publishing with the correct message properties and payload is the partner team application's responsibility. It is also the responsibility of each VRO microservice to validate requests before processing, or, in the case of a pass-through service like svc-bip-api, pass the request as-is to a downstream service and report the response as reported by the downstream service.

The Request / Response Model shall be implemented only on direct exchanges.

Requests

The following structure shall be used for publishing requests to VRO microservices via RabbitMQ:

Required Message Properties:

  • content_type="application/json"
  • app_id - name of calling application
  • reply_to - name of response queue if a response is desired
  • correlation_id - the string representation of a UUID assigned to a request
    • will also be returned with the response
    • used by the requestor to correlate each response to its original request

Required Payload:

The required payload is determined by the VRO microservice to which the requests are routed.

Responses

Required Message Properties:

  • content_type="application/json"
  • correlation_id - the string representation of a UUID used to correlate the request for which this response was made, if the request contained a correlation_id
  • app_id - id of application returning the response
  • other message properties set by RabbitMQ (see here)

Required Payload (aka. response body)

VRO microservices are responsible for determining the majority of the body of the response. The only required fields are the integer value statusCode and string value statusMessage for any type of request. The statusCode should represent a typical rest response code or the VRO microservice's defined values. The statusMessage should be a string representation of the statusCode for quick readability of the response.

If additional information regarding the status is desired, the VRO microservices is responsible for adding the optional field messages with statusCode and statusMessage. The messages field is an array of objects each containing the following fields:

  • key - required string - represents the error key identification
  • severity - required string - represents the severity of the error
  • status - optional integer - represents the HTTP status code
  • httpStatus - optional string - text representation of the response's HTTP status
  • text - optional string - text for more information
  • timestamp - optional string - ISO-8601 date-time of when error occurred

The messages construct is useful for services like the svc-bip-api where the response might be directly passed through another downstream service.

Example payloads:

Response for a request that does not have any additional fields in response body:

{
   "statusCode":201,
   "statusMessage":"CREATED"
}

Response for a request that resulted in a 500

{
   "statusCode":500,
   "statusMessage":"INTERNAL_SERVER_ERROR"
}

Response for a request that resulted in a 500 with additional error information included

{
   "statusCode":500,
   "statusMessage":"INTERNAL_SERVER_ERROR"
   "messages": [
      {
         "key":"java.class.where.error.happened"
         "severity":"FATAL"
         "status":500,
         "httpStatus":"INTERNAL_SERVER_ERROR",
         "text":"Something happened downstream..."
         "timestamp":"2023-12-04T12:35:12Z"
      }
   ]
}

Response for a request to a service that also returns an array of results:

{
   "statusCode":200,
   "statusMessage":"OK"
   "results":[
     {
        "fieldName":"thing1",
        "intVal":1
     },
     {
        "fieldName":"thing1",
        "intVal":1
     }
   ]
}

Stream / Push Model

TBD - be sure to mention "The Stream / Push Model shall be implemented only on fanout exchanges."

Partner Team Recommendations

Partner teams are welcome to implement their own solutions, provided that those solutions adhere to the information presented above. If there are any questions or concerns, contact the VRO team.

For parter teams implementing their application in python using the Request / Response Model, they can use a solution that follows this strategy by utilizing the hoppy library for asynchronous request/response patterns with a client that has configurable RabbitMQ connection parameters, retries and timeout policies, queue and exchange declarations, and more!

Additional Considerations: